Data Visualization
When Management Asks You To Produce an AI Magic
Using Plotly and Tableau to produce interactive graphs and dashboards to impress the management team.
Introduction
I was given quite simple and at the same time hard tasks by senior management. The data set contains five columns with two of them are categories and the other three were unique Identifiers.
So, I was left with only two columns to work with, they wanted some Ai product or something flashy and I have no idea how to produce flashy stuff. I never have dealt with such small data with few important features to work with. So, I started with CTGAN to generate more data from sample data, but it was not flashy, and tons of code will confuse more than solve the problem.
I have narrow down a few Ideas
- CTGAN Training and Generation
- Data Analysis on Deepnote
- Creating Visualization on Tableau
- Dashboarding
- Combining the simple data with larger available data
I narrow down it into simple visualization in Deepnote and integrating Tableau into Notebook.
Data Analysis
Importing pandas and Plotly for flashy visualization.
The Dataset contains five columns, we have dropped two of them due to privacy and irrelevancy. I have used Pec # as the index column so that we can focus on only two useful columns. You can see how simple the data is and how it's difficult for me to come up with an analysis.
Simple Histogram of Engineering Discipline
we can observe that computer and electrical engineering are taking the lead. In reality, civil, electrical, and mechanical engineering are in high demand on the other hand this data is sample data, and we are using it to build sample analysis so that we can just add original data.
Sunburst on Engineering Discipline
This was the first time I was using the Sunburst plot for categorical data. It’s quite interactive, if you click on any discipline, it will open up life flower and show you how many are employed or unemployed. I think I will be using this type of visualization quite often now.
A quick look at the data you can see Petroleum and Gas have more unemployed engineers and to simplify you can interact with the data to notice that every important engineering field have either private contract, or they are unemployed.
The simple Pie chart on Employment Sector
I have used Plotly pie chart and customize few things to look like a doughnut. It's clear that most of the engineers are either hired by private firms or are unemployed. This is a genuine problem in Pakistan as universities are pumping in more and more engineers every year and there are not enough firms to hire them and it’s creating a huge imbalance in the industry sector.
Tableau Visualization in Deepnote
At the same time, I started working on Tableau public and published the story of the initial visualization. At first, the whole thing was quite simple as it had bar charts and simple pie charts. I wanted to convert this all into Dashboard, so I started by adding all of my visualizations into the tableau platform and created this beauty. I think I have done justice for the work that was required by me. I couldn’t produce some magical Ai product but producing a working dashboard was quite fulfilling.
How to integrate public Dashboard into Deepnote?
It was quite simple and with few searches, I got a solution on Stack Overflow. You can just click on the share button at the bottom of your viz after saving your Dashboard on public.tableau.com. Then copy all embedded code.
Add %%HTML in your code cell and then paste embedded code and the whole dashboard will appear in your python notebook.
Closing Thoughts
I think we can do whole a lot more in the Jupyter notebook and to be honest I was surprised to see what I have created. The moral of the story is simple: keep looking for solutions even if things are not going according to your plan. I learned a lot in these few days, and it made me better at dealing with problems, even with the smaller set of data.
In the end, the management team was happy with my flashy Dashboard and visualization. They are now moving forward with a larger dataset, and they want me to build a data pipeline that will collect data from engineers and distributes critical information to a specific department so that other departments can make better decisions based on data.
You can follow me on LinkedIn and Polywork where I publish article every week.
Don’t forget to give your 👏
