You are reading the article Five Ways Data Science Has Evolved updated in December 2023 on the website Achiashop.com. We hope that the information we have shared is helpful to you. If you find the content interesting and meaningful, please share it with your friends and continue to follow and support us for the latest updates. Suggested January 2024 Five Ways Data Science Has Evolved
According to Figure Eight’s Annual Data Science Report, 89% of data scientists love their activity, up from 67% in 2023. 49% of data scientists get reached in any event once every week for a new job. Data Scientists are essentially more inclined by almost 75% to trust that AI will be great on the planet when compared with 39% of morals experts. A ton has changed since the organization’s unique Data Science Report in 2023. Machine learning ventures are increasing and an ever-increasing number of data is required to drive them. Data Science and machine learning employments are LinkedIn’s more quickly developing occupations. Also, the web is making 2.5 quintillion bytes of information every day to power every last bit of it. Until a couple of years back, just a bunch of us had known about
Data science is more applied than any time in recent memory Difficulties Involved while dealing with Noisy Datasets Knowledge of applied science winsUnderstanding the inside operations of the black box has turned out to be less imperative, except if you are the maker of the black box. Less data scientists with genuinely deep learning of statistical strategies are kept in the lab making the secret elements that ideally get coordinated within tools. This is to some degree baffling for long time data experts with thorough statistical foundation and understanding, however, this way might be important to genuinely scale modeling endeavors with the volume of information, business questions, and complexities we currently should reply.
Transition from Data-Poor to Data-RichAs organizations progress from data poor enterprises to data-rich, wide experience and an intensive foundation in both data science and pure sciences will be required. With institutes hurrying to overcome any issues and adjusting educational programs to current industry request, the supply gap will steadily diminish. However, as individuals in their late 20s, 30s and even 40s hope to turn towards a profession in data science, they ought to essentially expand on critical, applied learning and get genuine hands-on understanding. One can’t turn into a data analyst with only one analytics track or online accreditation, one needs to augment a solid applied statistics program. Hands-on experience can go far in clearing the most troublesome ideas of data science.
Data Science is both art and science Data science and statistics are interconnectedAs the field develops, the job of data scientists will evolve. One of the definitions being bandied around is that data scientists are experts in statistics. In any case, it may not be the situation with the current part which has floated from the engineering field. We have frequently heard that data science can’t be more than statistics. Sean Owen, Director of Data Science at Cloudera noticed that statistics and numerical processing have been associated for a considerable length of time, and as in every aspect of computing, we generally ache for approaches to analyse somewhat more data. As indicated by John Tukey’s paper The Future of Data Analysis, statistics must wind up worried about the dealing with and processing of data, its size, and perception. In any case, today many individuals from different background, even economics, guarantee to be data scientists. Truth be told, the research additionally spread out a couple of characterized occurrences where a portion of the data science-related tasks could turn out to be totally automated, robotized selection and tuning. The tasks that will end up being the center range of abilities later on are, highlight building and model approval, comprehension of the area, machine learning.
You're reading Five Ways Data Science Has Evolved
Top 6 Data Science Jobs In The Data
This data science career is doing very well on the market. Data science is making remarkable progress in many areas of technology, economy and commerce. It’s not an exaggeration. It is no surprise that data scientists will have many job opportunities.
It is true. Multiple projections show that the demand for data scientists will rise significantly in the next five-years. It is clear that demand is far greater than supply. Data science is a highly specialized field that requires a passion for math and analytical skills. This gap is perpetuated by the insufficient supply of these skills.
Every organization in the world is now data-driven. Data-driven organizations are the First Five: Google, Amazon, Facebook, Meta, Apple, Microsoft, and Facebook. They aren’t the only ones. Nearly every company in the market uses data-driven decision-making. The data sets can be customized quickly.
Amazon keeps meticulous records of all our choices and preferences in the world of shopping. It customizes the data to only send information that is relevant to the search terms of specific customers. Both the client and the company benefit from this process. This increases the company’s profit and helps the customer by acquiring goods at lower prices than they expected.
Data sets have a wider impact than just their positive effects. Data sets have positive effects on the health sphere by making people aware about critical health issues and other health-related items. It can also have an impact on agriculture, providing valuable information to farmers about efficient production and delivery of food.
It is evident that data scientists are needed around the globe, which makes their job prospects bright. Let’s take a look at some of the most exciting data science jobs available to data scientists who want to be effective in data management within organizations.
Top 6 Data Science Jobs in the Data-driven Industry 1. Data scientistsAverage Salary: US$100,000.
Also read: 14 Best Webinar Software Tools in 2023 (Ultimate Guide for Free)
2. Data architectsAverage Salary: US$95,000/annum
Roles and Responsibilities This employee is responsible for developing organizational data strategies that convert business requirements into technical requirements.
3. Data engineersAverage Salary: US$110,000 an Year
Also read: The 15 Best E-Commerce Marketing Tools
4. Data analystsAverage Salary: US$70,000 an Year
Roles and Responsibilities. A data analyst must analyze real-time data using statistical techniques and tools in order to present reports to management. It is crucial to create and maintain a database and analyze and interpret current trends and patterns within those databases.
5. Data storytellerAverage Salary: US$60,000 an Year
Also read: 10 Best Chrome Extensions For 2023
6. Database administratorsAverage Salary: US$80,000 an Year
Roles and Responsibilities of a database administrator: The database administrator must be proficient in database software to manage data effectively and keep it up-to date for data design and development. This employee will manage the database access and prevent loss and corruption.
These are only a few of the many data science jobs available to the world. In recent years, data science has been a thriving field in many industries around the globe. In this fast-paced world, data is increasingly valuable and there are many opportunities to fill data-centric roles within reputable organizations.
Data Science Roles In Telecom Industry
Introduction
Big Data and Cloud PlatformIn the early years, telecommunications data storage was hampered by a variety of problems such as unwieldy numbers, a lack of computing power, prohibitive costs. But with the new technologies, the dimension of problems has changed.
The areas of use of Technology are:
· Cloud Platform enabling Data storage expenses to drop every day. (Azure, AWS)
· Computer processing power is increasing exponentially (Quantum Computing)
· Analytics software and tools are cheap and sometimes free (Knime, Python)
In earlier days, the data stores were expensive, and data was stored in siloed – separated and often incompatible – data stores. This was creating barriers to make use of an enormous volume and variety of information. Business Intelligence (BI) vendors like IBM, Oracle, SAS, Tibco, and QlikTech are breaking down these walls between data storage and this provides a lot of jobs for telecom data scientists.
Data Scientist roles in Telecom Sector 1. Network OptimizationWhen a network is down, underutilized, overtaxed, or nearing maximum capacity, the costs add up
In the past, telecom companies have handled this problem by putting caption data and developing tiered pricing models.
But now, using real-time and predictive analytics, companies analyze subscriber behavior and create individual network usage policies.
When the network goes down, every department (sales, marketing, customer service) can observe the effects, locate the customers affected, andimmediately implement efforts to address the issue.
When a customer suddenly abandons a shopping cart, customer service representatives can soothe concerns in a subsequent call, text, oremail.
Building360-degree profile of Network using CDRs, Alarms, Network Manuals, TemIP, etc. gives a better overview of the network health.
Not only does this make happy customers, but it also improves efficiencies and maximizes revenue streams.
Telecoms also have the option to combine their knowledge of network performance with internal data (e.g., customer usage or marketing initiatives) and external data (e.g., seasonal trends) to redirect resources (e.g., offers or capital investments) towards network hotspots.
2. Customer PersonalizationLike all the industries, Telecom has much more scope to personalize the services such as value-added services, data packs, apps to recommend based on following the behavioral patterns of customers. Sophisticated 360-degree profiles of customers assembled from all below help to build personalized recommendations for customers.
Customer Behaviourvoice, SMS, and data usage patterns
video choices
customer care history
social media activity
past purchase patterns
website visits, duration, browsing, and search patterns.
Customer Demographicsage, address, and gender
type and number of devices used.
service usage
geographic location
This allows telecom companies to offer personalized services or products at every step of the purchasing process. Businesses can tailor messages to appear on the right channels (e.g., mobile, web, call center, in-store), in the right areas, and in the right words and images.
Customer Segmentation, Sentiment analysis, Recommendation engines for more apt products for the customers are the illustrative areas where Data scientists can help for improvements.
3. Customer RetentionDue to customer dissatisfaction in any of the areas such as poor connection/network quality, poor services, high cost of services, call drops, competitors, less personalization, customer churn. This means they jump from network to network in search of bargains. This is one of the biggest challenges confronting a telecom company. It is far more costly to acquire new customers than to cater to existing ones.
To prevent churn, data scientists are employing both real-time and predictive analytics to:
Combinevariables (e.g., calls made, minutes used, number of texts sent, average bill amount, the average return per user i.e.ARPU) to predict the likelihoodof change.
Know when a customer visits a competitor’s website changes his/her SIM or swaps devices.
Use sentiment analysis of social media to detect changes in opinion.
Target specific customer segments with personalized promotions based on historical behavior.
React to retains customers as soon as the change is noted.
Predictive models, clustering would be the ways to predict the prospective churners.
Implemented Solution ApproachUsing big data and python, I have developed the solution to find the upcoming network failure before it takes place. The critical success factor defined were:
· Identify and prioritize the cells with call drop issues based on rules provided by the operator.
· Based on rules specified, provide relevant indicative information to network engineers that might have caused the issue in the particular cell.
· Provide a 360-degree view of network KPIs to the network engineer.
· Build a knowledge management database that can capture the actions taken to resolve the problem and
· Update the CRs as good and bad, based on effectiveness in resolving the network issue
As a huge data was getting created, the database used was Hadoop -Big Insights.
Data transformation scripts were in spark.
And the neural network was the ML technique used to find out the system parameters when historically alarms (the indication of network failure) in the system got generated.
This information was fed as a threshold and once in the real scenario the parameters start approaching the threshold, the internal alert for those cell sites get generated for the Network engineer to focus on as preventive analytics.
Once the network engineer identifies the problem and solves it, it gets documented in the knowledge repository for future reference.
And when exactly a similar situation occurs, network the engineer will not get notification of internal alert but also steps to solve which is build using knowledge repository.
ConclusionThe reduction in process time, dropped call rate, the volume of (transient) issues handled by engineer, mean time to solve the problem, cost, people and increase in Revenue, customers, customer satisfaction, efficiency, and productivity of network engineers are the main area of any industry which Data scientists would be of help.
Various data generation sources under Telecom sectors are booming areas for Data Scientists to innovate, explore, value add, and help the provider to provide data-driven AI/ML solutions by preventive analytics, process improvements, optimizations, predictive analytics.
Related
How To Learn Data Science From Scratch
Data science is the branch of science that deals with the collection and analysis of data to extract useful information from it. The data can be in any form, be it text, numbers, images, videos, etc. The results from this data can be used to train a machine to perform tasks on its own, or it can be used to forecast future outcomes. We are living in a world of data. More and more companies are turning towards data science, artificial intelligence and machine learning to get their job done. Learning data science can equip you for the future. This article will discuss how to learn data science from scratch.
Why is data science important?You are always surrounded by zettabytes and yottabytes of data. Data can be structured or unstructured. It is important for businesses to use this data. This data can be used to:
visualize trends
reduce costs
launch new products and services
extend business to different demographics
Your Learning Plan 1. Technical SkillsWe will start with technical skills. Understanding technical skills will help you understand the algorithms with mathematics better. Python is the most widely used language in data science. There is a whole bunch of developers working hard to develop libraries in Python to make your data science experience smooth and easy. However, you should also polish your skills in R programming. 1.1. Python Fundamentals Before using Python to solve data science problems, you must be able to understand its fundamentals. There are lots of free courses available online to learn Python. You can also use YouTube to learn Python for free. You can refer to the book Python for Dummies for more help. 1.2. Data Analysis using Python Now we can move towards using Python in data analysis. I would suggest chúng tôi as the starting point. It is free, crisp and easy to understand. If you want a more in-depth knowledge of the topic, you can always buy the premium subscription. The price is somewhere between $24 and $49 depending on the type of package you opt for. It is always useful to spend some money for your future. 1.3. Machine Learning using Python The premium package for chúng tôi already equips you with the fundamentals of ML. However, there are a plethora of free resources online to acquire skills in ML. Make sure whichever course you follow, it deals with scikit-learn. Scikit-learn is the most widely used Python library for data science and machine learning. At this stage, you can also start attending workshops and seminars. They will help you gain practical knowledge on this subject. 1.4. SQL In data science, you always deal with data. This is where SQL comes into the picture. SQL helps you organize and access data. You can use an online learning platform like Codeacademy or YouTube to learn SQL for free. 1.5. R Programming It is always a good idea to diversify your skills. You don’t need to depend on Python alone. You can use Codeacademy or YouTube to learn the basics of R. It is a free course. If you can spend extra money, then I would say opt for the pro package for Codeacademy. It may cost you somewhere around $31 to $15
2. TheoryWhile you are learning about the technical aspects, you will encounter theory too. Don’t make the mistake of ignoring the theory. Learn the theory alongside technicalities. Suppose you have learned an algorithm. It’s fine. Now is the time to learn more about it by diving deep into its theory. The Khan Academy has all the theory you will need throughout this course.
3. MathMaths is an important part of data science. 3.1. Calculus Calculus is an integral part of this curriculum. Every machine learning algorithm makes use of calculus. So, it becomes inevitable to have a good grip on this topic. The topics you need to study under calculus are: 3.1.1. Derivatives
Derivative of a function
Geometric definition
Nonlinear function
3.1.2. Chain Rule
Composite functions
Multiple functions
Derivatives of composite functions
3.1.3. Gradients
Directional derivatives
Integrals
Partial derivatives
3.2. Linear Algebra Linear algebra is another important topic you need to master to understand data science. Linear algebra is used across all three domains – machine learning, artificial intelligence as well as data science. The topics you need to study under linear algebra are: 3.2.1. Vectors and spaces
Vectors
Linear dependence and independence
Linear combinations
The vector dot and cross product
3.2.2. Matrix transformations
Multiplication of a matrix
Transpose of a matrix
Linear transformations
Inverse function
3.3. Statistics Statistics are needed to sort and use the data. Proper organization and maintenance of data need the use of statistics. Here are the important topics under this umbrella: 3.3.1. Descriptive Statistics
Types of distribution
Central tendency
Summarization of data
Dependence measure
3.3.2. Experiment Design
Sampling
Randomness
Probability
Hypothesis testing
Significance Testing
3.2.3. Machine Learning
Regression
Classification
Inference about slope
4. Practical experienceNow you are ready to try your hands in some real-world data science problem. Enroll in an internship or contribute in some open-source project. This step will help you enrich your skills.
Data Science LifecycleEvery data science project goes through a lifecycle. Here we describe each of the phases of the cycle in detail.
Discovery: In this phase, you define the problem to be solved. You also make a report regarding the manpower, skills and technology available to you. This is the step where you can approve or reject a project.
Data Preparation: Here you will need to prepare an analytical sandbox that will be used in the remaining part of the project. You also need to condition the data before modeling. First, you prepare the analytical sandbox, then prepare ETLT, then data conditioning and finally visualization.
Model Planning: Here you will need to draw a relationship among the variables. You need to understand the data. These relationships will be the basis of the algorithm used in your project. You can use any of the following model planning tools: SAS/ACCESS, SQL or R.
Model Building: Here you need to develop data sets to train your system. You have to make a choice between your existing tools or a new more robust environment. Various model-building tools available in the market are SAS Enterprise Manager, MATLAB, WEKA, Statistica, Alpine Miner, etc.
Operationalize: In this step, you deliver a final report, code of the system and technical briefings. You also try to test the system in pilot mode to ascertain how it functions before deploying it in the real world.
Communicate Results: Now your work is done. In this step, you communicate with the stakeholders, whether or not your system complies with all their requirements ascertained in step 1. If they accept the system, your project is a success, or else it is a failure.
Data Science Components
Data: Data is the basic building block of data science. Data is of two types: structured data (is basically in tabular form) and unstructured data (images, emails, videos, PDF files, etc.)
Programming: R and Python are the most widely used programming language in data science. Programming is the way to maintain, organize and analyze data.
Mathematics: In the field of mathematics, you don’t need to know everything. Statistics and probability are mostly used in data science. Without the proper knowledge of mathematics and probability, you will most probably make incorrect decisions and misinterpret data.
Machine Learning: As a data scientist, you will be working with machine learning algorithms on a daily basis. Regression, classification, etc. are some of the well-known machine learning algorithms.
Big Data: In this era, raw data is compared with crude oil. Like we refine crude oil and use it to drive automobiles, similarly, the raw data must be refined and used to drive technology. Remember, raw data is of no use. It is the refined data that is used in all machine learning algorithms.
Now you know everything about data science. Now you have a clear road map on how to master data science. Remember this will not be an easy career. Data science is a very young market. Breakthrough developments are taking place almost every day. It is your job to keep yourself acquainted with all the happenings in the market. A little effort and a bright future await you.
About Author:Senior Data Scientist and Alumnus of IIM- C (Indian Institute of Management – Kolkata) with over 25 years of professional experience Specialized in Data Science, Artificial Intelligence, and Machine Learning. PMP Certified ITIL Expert certified APMG, PEOPLECERT and EXIN Accredited Trainer for all modules of ITIL till Expert Trained over 3000+ professionals across the globe Currently authoring a book on ITIL “ITIL MADE EASY” Conducted myriad Project management and ITIL Process consulting engagements in various organizations. Performed maturity assessment, gap analysis and Project management process definition and end to end implementation of Project management best practices Name: Ram Tavva Designation: Director of ExcelR Solutions Location: Bangalore
Pandas Cheat Sheet For Data Science In Python
What is Pandas Cheat Sheet?
Pandas library has many functions, but some of these are confusing for some people. We have here provided a helpful resource available called the Python Pandas Cheat Sheet. It explains the basics of Pandas in a simple and concise manner.
👉 Download the PDF of Cheat Sheet here
Whether you are a newbie or experienced with Pandas, this cheat sheet can serve as a useful reference guide. It covers a variety of topics, including working with Series and DataFrame data structures, selecting and ordering data, and applying functions to your data.
In summary, this Pandas Python Cheat Sheet is a good resource for anyone looking to learn more about using Python for Data Science. It is a handy reference tool. It can help you improve your data analysis skills and work more efficiently with Pandas.
Explaining important functions in Pandas:
To start working with pandas functions, you need to install and import pandas. There are two commands to do this:
Step 1) # Install PandasPip install pandas
Step 2) # Import PandasImport pandas as pd
Now, you can start working with Pandas functions. We will work to manipulate, analyze and clean the data. Here are some important functions of pandas.
Pandas Data StructuresAs we have already discussed that Pandas has two data structures called Series and DataFrames. Both are labeled arrays and can hold any data type. There is The only difference that Series is a one-dimensional array, and DataFrame is two-dimensional array.
1. SeriesIt is a one-dimensional labeled array. It can hold any data type.
s = pd.Series([2, -4, 6, 3, None], index=['A', 'B', 'C', 'D', 'E']) 2. DataFrameIt is a two-dimensional labeled array. It can hold any data type and different sizes of columns.
data = {'RollNo' : [101, 102, 75, 99], 'Name' : ['Mithlesh', 'Ram', 'Rudra', 'Mithlesh'], 'Course' : ['Nodejs', None, 'Nodejs', 'JavaScript'] } df = pd.DataFrame(data, columns=['RollNo', 'Name', 'Course']) df.head() Importing DataPandas have the ability to import or read various types of files in your Notebook.
Here are some examples given below.
# Import a CSV file pd pd.read_csv(filename) # Import a TSV file pd.read_table(filename) # Import a Excel file pd pd.read_excel(filename) # Import a SQL table/database pd.read_sql(query, connection_object) # Import a JSON file pd.read_json(json_string) # Import a HTML file pd.read_html(url) # From clipboard to read_table() pd.read_clipboard() # From dict pd.DataFrame(dict) SelectionYou can select elements by its location or index. You can select rows, columns, and distinct values using these techniques.
1. Series # Accessing one element from Series s['D'] # Accessing all elements between two given indices s['A':'C'] # Accessing all elements from starting till given index s[:'C'] # Accessing all elements from given index till end s['B':] 2. DataFrame # Accessing one column df df['Name'] # Accessing rows from after given row df[1:] # Accessing till before given row df[:1] # Accessing rows between two given rows df[1:2] Selecting by Boolean Indexing and Setting 1. By Position df.iloc[0, 1] df.iat[0, 1] 2. By Label df.loc[[0], ['Name']] 3. By Label/Position df.loc[2] # Both are same df.iloc[2] 4. Boolean Indexing# Use filter to adjust DataFrame
# Set index a of Series s to 6 s[‘D’] = 10 s.head()
Data CleaningFor data cleaning purposes, you can perform the following operations:
Rename columns using the rename() method.
Update values using the at[] or iat[] method to access and modify specific elements.
Create a copy of a Series or data frame using the copy() method.
Check for NULL values using the isnull() method, and drop them using the dropna() method.
Check for duplicate values using the duplicated() method. Drop them using the drop_duplicates() method.
Replace NULL values using the fill () method with a specified value.
Replace values using the replace() method.
Sort values using the sort_values() method.
Rank values using the rank() method.
# Renaming columns df.columns = ['a','b','c'] df.head() # Mass renaming of columns df = df.rename(columns={'RollNo': 'ID', 'Name': 'Student_Name'}) # Or use this edit in same DataFrame instead of in copy df.rename(columns={'RollNo': 'ID', 'Name': 'Student_Name'}, inplace=True) df.head() # Counting duplicates in a column df.duplicated(subset='Name') # Removing entire row that has duplicate in given column df.drop_duplicates(subset=['Name']) # You can choose which one keep - by default is first df.drop_duplicates(subset=['Name'], keep='last') # Checks for Null Values s.isnull() # Checks for non-Null Values - reverse of isnull() s.notnull() # Checks for Null Values df df.isnull() # Checks for non-Null Values - reverse of isnull() df.notnull() # Drops all rows that contain null values df.dropna() # Drops all columns that contain null values df.dropna(axis=1) # Replaces all null values with 'Guru99' df.fillna('Guru99') # Replaces all null values with the mean s.fillna(s.mean()) # Converts the datatype of the Series to float s.astype(float) # Replaces all values equal to 6 with 'Six' s.replace(6,'Six') # Replaces all 2 with 'Two' and 6 with 'Six' s.replace([2,6],['Two','Six']) # Drop from rows (axis=0) s.drop(['B', 'D']) # Drop from columns(axis=1) df.drop('Name', axis=1) # Sort by labels with axis df.sort_index() # Sort by values with axis df.sort_values(by='RollNo') # Ranking entries df.rank() # s1 is pointing to same Series as s s1 = s # s_copy of s, but not pointing same Series s_copy = s.copy() # df1 is pointing to same DataFrame as df df1 = s # df_copy of df, but not pointing same DataFrame df_copy = df.copy() Retrieving InformationYou can perform these operation to retrieve information:
Use shape attribute to get the number of rows and columns.
Use the head() or tail() method to obtain the first or last few rows as a sample.
Use the info(), describe(), or dtypes method to obtain information about the data type, count, mean, standard deviation, minimum, and maximum values.
Use the count(), min(), max(), sum(), mean(), and median() methods to obtain specific statistical information for values.
Use the loc[] method to obtain a row.
Use the groupby() method to apply the GROUP BY function to group similar values in a column of a DataFrame.
1. Basic information # Counting all elements in Series len(s) # Counting all elements in DataFrame len(df) # Prints number of rows and columns in dataframe df.shape # Prints first 10 rows by default, if no value set df.head(10) # Prints last 10 rows by default, if no value set df.tail(10) # For counting non-Null values column-wise df.count() # For range of index df df.index # For name of attributes/columns df.columns # Index, Data Type and Memory information df.info() # Datatypes of each column df.dtypes # Summary statistics for numerical columns df.describe() 2. Summary # For adding all values column-wise df.sum() # For min column-wise df.min() # For max column-wise df.max() # For mean value in number column df.mean() # For median value in number column df.median() # Count non-Null values s.count() # Count non-Null values df.count() # Return Series of given column df['Name'].tolist() # Name of columns df.columns.tolist() # Creating subset df[['Name', 'Course']] # Return number of values in each group df.groupby('Name').count() Applying Functions # Define function f = lambda x: x*5 # Apply this function on given Series - For each value s.apply(f) # Apply this function on given DataFrame - For each value df.apply(f) 1. Internal Data Alignment # NA values for indices that don't overlap s2 = pd.Series([8, -1, 4], index=['A', 'C', 'D']) s + s2 2. Arithmetic Operations with Fill Methods # Fill values that don't overlap s.add(s2, fill_value=0) 3. Filter, Sort and Group ByThese following functions can be used for filtering, sorting, and grouping by Series and DataFrame.
# Filter rows where column is greater than 100 # Filter rows where 70 < column < 101 # Sorts values in ascending order s.sort_values() # Sorts values in descending order s.sort_values(ascending=False) # Sorts values by RollNo in ascending order df.sort_values('RollNo') # Sorts values by RollNo in descending order df.sort_values('RollNo', ascending=False) Exporting DataPandas has the ability to export or write data in various formats. Here are some examples given below.
# Export as a CSV file df df.to_csv(filename) # Export as a Excel file df df.to_excel(filename) # Export as a SQL table df df.to_sql(table_name, connection_object) # Export as a JSON file df.to_json(filename) # Export as a HTML table df.to_html(filename) # Write to the clipboard df.to_clipboard() Conclusion:Pandas is open-source library in Python for working with data sets. Its ability to analyze, clean, explore, and manipulate data. It is an important tool for data scientists. Pandas is built on top of Numpy. It is used with other programs like Matplotlib and Scikit-learn. Pandas Cheat Sheet is a helpful resource for beginners and experienced users. It covers topics such as data structures, data selection, importing data, Boolean indexing, dropping values, sorting, and data cleaning. We have also prepared pandas cheat sheet pdf for article. Pandas is a library in Python and data science uses this library for working with pandas dataframes and series. We have discussed various pandas commands in this cheatsheet.
Colab of Cheat SheetMy Colab Exercise file for Pandas – Pandas Cheat Sheet – Python for Data Science.ipynb
A Quick Tutorial On Clustering For Data Science Professionals
This is article was published as a part of the Data Science Blogathon.
Welcome to this wide-ranging article on clustering in data science! There’s a lot to unpack so let’s dive straight in.
In this article, we will be discussing what is clustering, why is clustering required, various applications of clustering, a brief about the K Means algorithm, and finally in detail practical implementations of some of the applications using clustering.
Table of Contents
What is Clustering?
Why is Clustering required?
Various applications of Clustering
A brief about the K-Means Clustering Algorithm
Practical implementation of Popular Clustering Applications
What is Clustering?In simple terms, the agenda is to group similar items together into clusters, just like this:
Let’s go ahead and understand this with an example, suppose you are on a trip with your friends all of you decided to hike in the mountains, there you came across a beautiful butterfly which you have never seen before. Further, you encountered a few more. They are not exactly the same but similar enough for you to understand that they belong to the same species. Now here you need a lepidopterist(the one who studies and collects butterflies) to tell you exactly what species they are, but there is no need for an expert to identify a similar group of items. This way of identifying similar objects/ items is known as clustering.
Why is Clustering required?So Clustering is an unsupervised task. Unsupervised means the ones in which we are not provided with any assigned labels or scores for training our data.
Here in the above figure on the left, we can see that each instance is marked with different markers which means it’s a labeled dataset for which we can use the classification algorithms like SVM, Logistics Regression, Decision Trees, or Random Forests. On the right side if you observe it is the same dataset but without labels so here the story for classifications algorithms ends(i.e we can’t use them here). This is where the clustering algorithms come into the picture to save the day!. Right now in the above picture, it is pretty obvious and quite easy to identify the three clusters with our eyes, but that we not be the case while working with real and complex datasets.
Various applications of Clustering 1. Search engines:You may be familiar with the concept of image search which Google provides. So what this system does is that first, it applies the clustering algorithm on all the images available in the database available. After which similar images would fall under the same cluster. So when a particular user provides an image for reference what it will be doing is applying the trained clustering model on the image to identify its cluster once this is done it simply returns all the images from this cluster.
2. Customer Segmentation:We can also cluster our customers based on their purchase history and their activity on our website. This is really important and useful to understand who our customers are and what they require so that our system can adapt to their requirements and suggest products to each respective segment accordingly.
3. Semi-supervised Learning:When you are working on semi-supervised learning in which you are only provided with a few labels, there you could perform clustering algorithms and generate labels for all instances falling under the same cluster. This technique is really good for increasing the number of labels after which a supervised learning algorithm can be used and its performance gets better.
4. Anomaly detection:Any instance that has a low affinity(Measure of how well an instance fits into a particular cluster) is probably an anomaly. For example, if you have clustered the user based on the request per minute on your website, you can detect users with abnormal behavior. So this technique is particularly useful in detecting any manufacturing detects or for some fraud detections.
5. Image Segmentation:If you cluster all the pixels according to their colors, then after that we can replace each pixel with the mean color of its cluster, this might be helpful whenever we need to reduce the number of different colors in the image. Image segmentation plays an important part in object detection and tracking systems.
We will look at how to implement this further.
A Brief About the K-Means Clustering AlgorithmLet’s go ahead and take a quick look at what the K-means algorithm really is.
Firstly, let’s generate some blobs for a better understanding of the unlabelled dataset.
import numpy as np from sklearn.datasets import make_blobs blob_centers = np.array( [[ 0.2, 2.3], [-1.5 , 2.3], [-2.8, 1.8], [-2.8, 2.8], [-2.8, 1.3]]) blob_std = np.array([0.4, 0.3, 0.1, 0.1, 0.1]) X, y = make_blobs(n_samples=2000, centers=blob_centers, cluster_std=blob_std, random_state=7)Now let’s plot them
plt.figure(figsize=(8, 4)) plt.scatter(X[:, 0], X[:, 1], c=None, s=1) save_fig("blobs_plot") plt.show()So this is how an unlabeled dataset would look like, here we can clearly see that there are five blobs of instances. So basically k means is just a simple algorithm capable of clustering this kind of dataset efficiently and quickly.
Let’s go ahead and train a K-Means on this dataset. Now, this algorithm will try to find each blob’s center.
from sklearn.cluster import KMeans k = 5 kmeans = KMeans(n_clusters=k, random_state=101) y_pred = kmeans.fit_predict(X)Keep in mind that we need to specify the number of cluster k that the algorithm needs to find. In our example, it is pretty straight forward but in general, it won’t be that easy. Now after training each instance would have been assigned to one of the five clusters. Remember that here an instance’s label is the index of the cluster, don’t confuse it with class labels in classification.
Let’s take a look at the five centroids the algorithm found:
kmeans.cluster_centers_These are the centroids for clusters with indexes of 0,1,2,3,4 respectively.
Now you can easily be able to assign new instances and the model will assign it to a cluster whose centroid is closet to it.
new = np.array([[0, 2], [3, 2], [-3, 3], [-3, 2.5]]) kmeans.predict(new)That is pretty much it for now, we will see in detail working and types of K-Means some other day in some other blog. Stay Tuned!
Implementation of Popular Clustering Applications1. Image Segmentation using clustering
Image Segmentation is just the task of partitioning an image into multiple segments. For example, in a self-driving car’s object detection system, all the pixels that are part of a traffic signal’s image might be assigned to the “traffic-signal” segment. Today there are state of the art model based on CNN(convolution neural network) using complex architecture are being used for image processing. But we are going to do something much simpler which is color segmentation. We will simply assign pixels to a particular cluster if they have the same color. This technique might be sufficient for some applications, like the analysis of satellite images to measure the forest area coverage in a region, color segmentation might just do the work.
Let’s go ahead a load the image we are about to work on:
from matplotlib.image import imread image = imread('lady_bug.png') image.shapeNow Let’s go ahead and reshape the array to get a long list of RGB colors and then cluster them using K-Means:
X = image.reshape(-1, 3) kmeans = KMeans(n_clusters=8, random_state=101).fit(X) segmented_img = kmeans.cluster_centers_[kmeans.labels_] segmented_img = segmented_img.reshape(image.shape)Now what’s happening here is, for example, it tries to identify a color cluster for all shades of green. After that, for each color, it looks for the mean color of the pixel’s color cluster. What I mean is it will replace all shades of green with a light green color assuming that the mean is light green. At last, it will reshape this long list of colors to the original dimension of the image.
Output with a different number of clusters:
2. Data preprocessing using Clustering
For Dimensionality reduction clustering might be an effective approach, like a preprocessing step before a supervised learning algorithm is implemented. Let’s take a look at how we can reduce the dimensionality of the famous MNIST dataset using clustering and how much performance difference we get after doing this.
MNIST dataset consists of 1797 grayscale(one channel) 8 X 8 images representing digits from 0 to 9. Let’s start by loading the dataset:
from sklearn.datasets import load_digits X_digits, y_digits = load_digits(return_X_y=True)Now let’s split them into training and test set:
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X_digits, y_digits, random_state=42)Now let’s go ahead and train a logistic regression model and evaluate its performance on the test set:
from sklearn.linear_model import LogisticRegression log_reg = LogisticRegression() log_reg.fit(X_train, y_train)Now Let’s evaluate its accuracy on the test set:
log_reg_score = log_reg.score(X_test, y_test) log_reg_scoreOk so now we have an accuracy of 96.88%. Let’s see if we can do better by using K-Means as a preprocessing step. We will be creating a pipeline that will first cluster the training set into 50 clusters and replace those images with their distances to these 50 clusters, then after that, we will apply the Logistic Regression model:
from sklearn.pipeline import Pipeline
pipeline = Pipeline([
(“kmeans”, KMeans(n_clusters=50)),
(“log_reg”, LogisticRegression()),
])
pipeline.fit(X_train, y_train)
Let’s evaluate this pipeline on test set:
pipeline_score = pipeline.score(X_test, y_test)
pipeline_score
Boom! We just increased the accuracy of the model. But here we choose the number of clusters k arbitrarily. Let’s go ahead and apply grid search to find a better k value:
param_grid = dict(kmeans__n_clusters=range(2, 100)) grid_clf = GridSearchCV(pipeline, param_grid, cv=3, verbose=2) grid_clf.fit(X_train, y_train)Warning the above step might be time-consuming!
Let’s see the best cluster that we got and its accuracy:
grid_clf.best_params_The accuracy now is:
grid_clf.score(X_test, y_test)Here we got a significant boost in accuracy compared to earlier on the test set.
End NotesTo sum up, in this article we saw what is clustering?, why is clustering required? , various applications of clustering, a brief about the K Means algorithm, and lastly in detail practical implementations of some of the applications using clustering. I hope you liked it!
Stay tuned!
Connect me with on LinkedIn
Thank You!
The media shown in this article is not owned by Analytics Vidhya and are used at the Author’s discretion
Related
Update the detailed information about Five Ways Data Science Has Evolved on the Achiashop.com website. We hope the article's content will meet your needs, and we will regularly update the information to provide you with the fastest and most accurate information. Have a great day!