How Data Science Project Works - From the Koobiyo Teledrama

About five years ago, the Koobiyo teledrama was very popular due to the uncharacteristic nature of the teledrama. It was a political teledrama, one reason the teledrama became popular. However, this post is not to discuss the political side of the tele drama but to discuss the data science side of it. 

Jehan, the main character of this teledrama, has the idea of building a tool to predict the future problems of people. He uses decision tree architecture to achieve this and explains his idea to his companion, Priyantha as shown in the below video. Watch from the 540th second.

So his idea is very clear. By using data like Gender, Marital Status, Professional Qualification, Parents' details, Friends' details, etc. 

After building the predictive software, he presents his case to his good friend Hiruni and his developer too joins the discussion in the following video. Watch from the 40th second.

After observing the work done by Jehan, Hiruni was impressed, but she had a very vital question to ask. "Why do you think people will buy this". Jehan was a little perplexed by the question, and he stressed that this is the software that everyone will have at their fingertips. Obviously, people know their problems they need solutions, not to suffer from their own problems. 

This is something important to us as data scientists. We are very good at technical details, but we need to understand the solution part of the tool. Remember, people need a solution to their problem not a technical tool. 

Then Hiruni provide a solution to Jehan and asked him to provide this to a human resource person to predict the future problems of its own employees. 

Then the solution can be viewed in the 21st second of the following video.

In the live demo, Jehan proves that it is important to select the best people for the organization from this software. It is clear how we should think from the business point of view rather than focusing on technical aspect mainly. As a data scientist, we need to look at the solution side of it.

Article : Learn about Data Warehousing for Analytical Needs

I started another article series at www.mssqltips.com on data warehousing. Even though there are many tools to build data warehouses, I still find there is a lack of conceptual knowledge in the community. So trying to cover that gap. So, read this What is a Data Warehouse? article and send your comments. 

Who is a Data Scientist? A Dragon a Pegasus or a Unicorn?

 

Unicorns, Pegasus, or dragons are mythical creatures that are created by combining the strong attributes of different animals. In Sri Lankan culture, we can see different dragons in temples. It is believed that these dragons are to protect various people and places. 

These dragons have seven parts from different people, as expressed in the following poem. 

As per the song, the dragon is a combination of

·       trunk of an elephant

·       legs of a lion

·       ears of a pig

·       teeth of a crocodile

·       eyes of a monkey

·       body of a fish

·       wings of a bird

The combination of these most strengthened parts will make the dragon a strong animal to achieve his required tasks. 

Isn't that the same with a data scientist? Don't they need different strengths, as shown below?  

Does that mean data scientists are mythical? Read the full article here E-Magazine – Magazine that bridges Management and IT

Clustering as Pre-Processing Technique for Classification

Clustering is often used to identify natural groups in a dataset. Since the clustering technique does not depend on any independent variable, the clustering technique is said to be unsupervised learning. The classification technique is supervised as it models data for a target or dependent variable. This post describes clustering as a pre-processing task for classification. This post has used the Orange Data Mining tool to demonstrate the above scenario. 

Following is the complete orange data mining workflow, and this is available in the Git Hub as well. 

The attrition sample dataset is used to demonstrate the above scenario. This dataset contains 18 variables with 1470 instances. Then the Jaccard distance is used, and it is essential to remember that you may have to choose a different distance technique in order to improve the results. 

Then hierarchical clustering is used for three clusters as too many clustering will have smaller clusters and may cause overfitting. Following is the cluster distribution for the three clusters. 

After the clustering is completed, the dataset is selected for each cluster and unnecessary columns (Clsuter and Selected) are removed.  Then for each cluster, different classification techniques were executed. This scenario uses SVM, Logistic Regression, Neural Network, Random Forest, AdaBoost, Naive Bayes, Tree, and kNN classification techniques. 

When classification techniques were executed for the entire dataset, without clustering, the best classification technique was Logistic Regression with a Classification accuracy of 87%.  

Following are the classification accuracies and other evaluation parameters for each cluster once it is evaluated with all the above said eight different classification techniques.

The above table shows that C1 and C2 clusters have higher accuracy than the full dataset. However, the third cluster performance is not as good as the overall dataset classification.

This technique is better when there are large volumes of data if not clustering will reduce the data which can lead to model overfitting. 

Is GPT-4 a Job Destroyer?

With rise of GPT-4, many jobs will be on the firing line. For example, with the expectation of 10% letter writing will be done with the use of GPT-4, job of secaratries are in danger. What about the software developers.You you can upload a image and get the source code from the GPT-4!!!

Wouldn't that be the same fear we had when the computers were introduced. Being in the Software field in early 1990s in Sri Lanka, I still remember how much pressure we had from many employees when we try to install new software applications. During most of the product demostration, we used to conclude our sessions by addressing that fear. We said that new software system will not destroy any jobs but instead it will create new jobs. Isn't that true? If I look back the those industries back, yes some jobs have gone for good, but there are new jobs. 

Let us look at Cricket. During 1980's there were only a couple of camera for TV coverage. With the improvement in technology, now there are many cameras including spider, stump, drone etc. With these technologies, viewship has increased and more revenue avenues are being created. Further, cricket has become more glamoours and into a new world. With that new jobs are created and the good example is the third umpire.

Same will be inline for GPT. Let us wait and see in time to come.

2nd Evaluation of Oxford Mathematical Model during the Knockouts

We are into the business end of the sporting event, FIFA 2022. As we have been doing for last couple of weeks, let us evaluate the Oxfor MAthematical Model. Even though the much anticipated semi-finals are yet to be kicked off, we can do the full evaluation. It is because, even now we know that the oxford predication for the final wont be a correct, as Brazil and belgium are out of the competition. 

Let us analysis the stage wise accuracy rates. 

Problem with this accuracy calculation is that the error is propagated. Since Oxford predicted that the Belgium will feature in the final in which they were out of the competition from the first round, that is reflected at every stage. 

Would you accept this model with just over 50% of accuracy. That means it is just over the ad-hoc  probability prediction. As we have been insisting, predicting the results outcome of a sport event, is not yet successful, it least for the moment. 

Evaluation of Oxford Mathematical Model after First Round

We have been discussing the FIFA results prediction that was developed by Oxford University. Since the first round is completed, we can now do a model evaluation with the actual data vs. predicted data. Oxford mathematical model has two components, predicting the teams for the next round and the teams' position at each group. 

As we saw in FIFA 2022, there were many surprises. While Japan, Australia, Morocco, and South Korea advanced to the next round, we saw Belgium (The number 2 ranked team), four times champions, and Germany out of the world cup in the first round. 

Following is the final 16s.

Out of the Oxford predictions for the next round, 9 out of 16 are correct, which is 65% of accuracy. Let us look at the group-wise rankings. Out of the 32 positions, only 11 were correct, meaning the accuracy is less than 35%. 

When the group-wise analysis was made. All Group F position predictions were wrong, while there are no groups that all positions are correct. 

When the position analysis was made, there were a few notable observations.

We can see a higher accuracy for positions 1 and 4. That means it is easy to predict the best and poorest teams. This is not again a hard part. However, their model could be better when predicting the number 2 and 3 positions. In fact, the number 3 position prediction accuracy is 9%. 

Sport is all about fractions, inches, and moments. That is where the prediction of results is not difficult but impossible. If it is possible, it is not a sport.

Prediction of FIFA results went wrong, AGAIN!

No matter what level of mathematical models you can build, still predicting the result of a sport is extremely difficult, if not impossible. FIFA 2022 started with a lot of predictions, and we are just into the  4th day and 12th match, yet we have watched the two greatest upsets in the FIFA world cup.

The first, I would say it is the greatest upset of all time in which Saudi Arabia won against Messi's Argentina.

At the end of the first half, Argentina was leading 1-0, but in the next half, the oil nation scored two goals quickly. After the Saudi's first goal, the Argentinian giants were in disarray. In the end, most of the analytics said that crowd support for Saudi Arabia was one of the reasons for their surprising win.  

The second match was between Japan and four times world cup winners (1954, 1974, 1990, 2014) Germany. 

Germany was showing their class during the first half and scored the first goal. Germany has never lost a world cup match after 1954 after scoring the first goal. On the other hand, Japan has never won a world cup match after conceding a first goal. Even though statistics were not on their side, Japan won the match against Germany with a score line of 2-1. After the match, the analytics said that most of the Japanese players had played in the German football cup, which gave them an advantage over Germany. 

How can we include crowd support and players' special features into mathematical modeling? I would say it is difficult, which is why sport is such an unpredictable and, thus, fantastic event. We can enjoy sports events only because of their unpredictable nature. If not, sport is not a sport, and let the sport be a sport. 

Who is the Next FIFA Champion?

After the Olympics, the world's most famous sporting event will raise its curtain today at Quater. The FIFA 2022 will be kicked off with 32 teams, and the obvious question would be "Who will be the winner". Will it be Brazil, Netherlands, Spain, Argentina, France, England, Belgium, or any other dark horses?

If you remember in FIFA 2010, Paul the Octopus predicted all the winners of the matches that involved Germany and the final between Spain and the Netherlands with an accuracy of 100%. 

Paul the Octopus made friends and enemies on his own due to the predictions he made. Now in 2022, the University of Oxford has come up with a mathematical model to predict the winner. 

The following figure is its prediction from the round of 16s. 

According to the Oxford model, Brazil is the ultimate winner. However, looking at the recent, it will be a surprise. 

Further, after 1998 defending champions did not perform well in the world cup finals. So France to be in the semi-finals is something against recent data. 

I personally do not believe that predictions can be made for sports concerning its outcome. If you do that, I believe that will be the end of sports. Sport is beautiful because of the extremely high uncertainty. One touch or one move or one wrong decision may change the cause of the game drastically. In 2002 FIFA saw a lot of such matches. Who would have expected the defending champion, France to be beaten by the African nation Senegal? Who would have thought France would be out of the world cup in the first round without any wins? Among all the surprises, South Korea made the headlines by reaching the Semifinals and becoming the giant killer of Poland, Portugal, Italy, and Spain.

Let the sport be a sport!

Predictions Went Wrong

Predictions are part and parcel of our life. With more available data, we are looking at driven predictions rather than sense-driven predictions. Predication has expanded many domains such as ecomics, productions, sports etc. However, last week, we saw two of such predictions went wrong. 

1. Whether predictions at T20I worldcup

First of all, pre-world cup favorites, India couldn't win the semi final while another pre tournmennt favorite, hosts, Aurtralia couldn't make even to the semi final. I am not going to consider that a wrong predictions as win/loss prediction in sports should fail in order to sport become more attractive. 

However, when England - Pakistan lock horns with each other at the Melbourne for the final, there were predictions that match will not be possible to stage due to rain. There was 95% probability for rain. Playing conditions were altered in order to accommodate rain. Additonal day was already announced even before the tournament. But with the prediction of rain, match duration was increased by two hours and ten overs were set to constitute a match.  However, match was played without any interruptions. There was a harmless drizzle for five to ten minutes.

What was Predicted

What was Actual

2. USA Mid Term Elections

Then the next prediction in USA mid term elections. It was expected to trump led republican to take over the senate, house and even the govenership easily. There was prediction of "red - wave" and trump's come back.. But at the end of the election, demoract was able to hold the power of senete with 50/50. (With the vice president twi breaker, they are narrowly holding the senete by one vote. Predications were expecting republicans to win 55 seats in senete. On the House, republicans were expecting to get 235 seats but they got 218 which is the cut of margin to win the state. (five seats yet to be called). Now many media organizations are calling to review the pre-election predictions.

Source: US midterm elections results 2022: live | US midterm elections 2022 | The Guardian

Finaly,we need to understand that every thing under the sun cannot be predicted even with the sofiticated technologies that are around.