ml fixation on task performance

content/notes/the machine learning industry is too focused on general task performance.md

@@ -0,0 +1,8 @@
+
+The machine learning industry has traditionally adopted an academic approach, focusing primarily on performance across a range of tasks. Large Language Models (LLMs) like GPT-4 are a testament to this approach, having been scaled up to demonstrate impressive capabilities across numerous tasks. This scaling has led to the emergence of new abilities, although debates about the true nature of these emergent abilities continue.
+
+However, the issue that arises with this approach is that while these models are generally capable, they may not perform tasks in the way an individual user would prefer. This is a failure mode that anyone building agents will inevitably encounter. The focus, therefore, needs to shift from how language models perform tasks in a general sense to how they perform tasks on a user-specific basis.
+
+Take the task of summarization as an example. It’s a popular machine learning task and models have become quite proficient at it, at least from a benchmark perspective. However, when these models summarize for users, the results often fall short. The reason for this is simple: the models don’t summarize things the way an individual user would. The key takeaways for a user would differ from the takeaways of the average internet user.
+
+Therefore, a shift in focus towards user-specific task performance would provide a much more dynamic and realistic approach. This would not only cater to the individual needs of the user but also pave the way for more personalized and effective machine learning applications.