Anyone who works with data knows how crucial performance is, especially when performing complex data processing and data transformation operations on medium to large datasets.

At Rulex, we understand this need very well, which is why we have devoted a considerable amount of time and effort to ensuring that our software is incredibly fast and efficient.

Processing data fast

Rulex Platform is optimized to handle complex data operations at scale with lightning-fast speed, ensuring that users can process their data quickly and responsively. This feature is especially crucial for companies that rely on near real-time data analytics in their decision-making processes, as slow performance levels can lead to delays and inaccurate information, ultimately impacting services, resources, and business decisions.

Data processing speed: Rulex vs Pandas

To showcase the fast data processing capabilities of Rulex, we have compared it with Pandas, an open-source data manipulation library built on top of the Python programming language.

However, while Pandas is a powerful tool, it can struggle when handling large datasets or complex data operations, leading to slower processing times.

Rulex Platform handles these challenges with speed and efficiency, making it an excellent choice for businesses that need to process data quickly and accurately.

To provide an accurate comparison of Rulex Platform and Pandas, we conducted a series of tests using identical conditions on the same machine and measured the results. We performed ten different operations (group, filter, sort, join, math calculations, concatenation and a sequence of operations) on datasets with the following characteristics: an initial relatively small dataset with 5 million rows of data, a second medium-sized dataset with 15 million rows of data and a final large dataset with 50 million rows of data.

Performance results

Here is a brief summary of our findings to give you an idea of the results we obtained.

More Rulex-Panda data performance comparison

If you are interested in learning more about our testing methodology and results, we have provided a detailed description on Rulex Community: Rulex Platform vs Pandas: Performance Comparison.

Feel the speed of Rulex Platform

Interested in trying Rulex Platform straightway? Get a 30-day free trial.

We all know that data-driven decision-making processes can bring huge benefits to companies, such as greater confidence in taking decisions, more transparency, and a higher chance of improving in the long term.

Gartner estimates that more than 33% of large organizations will have analysts practicing decision intelligence by 2023, and 75% of enterprises will shift from piloting to operationalizing AI, driving a 5X increase in streaming data and analytics infrastructures by the end of 2024.

But at the same time, businesses often have doubts on how to deliver data-driven decision-making projects successfully.

In a previous article, we talked about how important it is to choose the right project when implementing a data-driven decision-making process. We said that businesses should think about the goals they want to achieve, consider the available data, and the actionable solutions that might be implemented thanks to their data-driven knowledge.

Let’s now assume we have identified the project we want to put in production.

HOW TO STRUCTURE DATA-DRIVEN DECISION-MAKING PROJECTS

Once the project team has been created – it should be balanced, well defined, and with all the stakeholders involved – we advise you to include 3 phases:

• Project Scope (<1 week)
• Pilot (between 4 and 6 iterations, each iteration lasts for one week)
• Production deployment (between 2 and 4 weeks)

The phases can be tailored according to your needs but, in principle, we recommend you follow them as closely as possible.

Let’s imagine your data-driven decision-making project is highly complex and requires many months of work to get to the final results.

We suggest you reduce the initial scope to give you a first deliverable in 2 or 3 months.

This has two major advantages:

• You focus your efforts on something immediately usable
• You build trust in all stakeholders, who see results from the outset.

One of the advantages of Rulex is that it is extremely flexible and interactive thanks to its drag and drop functionality and eXplainable AI. As a result, it delivers value from the very first iterations.

The graph below sums up the life cycle of every Rulex project: at an early stage, little effort is involved and the return in value is already greater than the investment.

Over time, the management effort grows slowly, while the value of the solution grows exponentially.

HOW TO DEFINE YOUR PROJECT SCOPE

The scope phase is crucial to the success of any data-driven decision-making project.

If you make a mistake in this phase, for instance, setting an incorrect goal or misunderstanding your data, the project might be seriously compromised.

Typical activities during the assessment phase are:

• Goal definition: set out the goal in detail, identifying performance measures and expected KPIs
• Data definition and audit: identify the available data, evaluating data quality and suitability for analysis
• Deployment in production: plan how the new decision-flow will be deployed in production, e.g., how it will take place within the daily work routine and how it will interact with all company processes
• Assessment: evaluate whether objectives can be met or if corrective actions should be taken

If the assessment is positive, project sponsors should sign-off so you can move to the pilot phase.

DEVELOP YOUR PILOT, FOCUSING ON VALUE CREATION

In the pilot phase, the team should provide proof of concept, and clients should see the first results. With conventional methods, customers expect these steps to take several months, requiring a large number of billable hours from data scientists, and delivering little or no directly actionable insights.

With Rulex, customers will quickly receive tangible business value in the form of understandable, actionable insights.

In this phase, typical activities are data preparation, model creation and refining, and dashboard creation.

DEPLOY IN PRODUCTION FAST AND SUCCESSFULLY

In the final stage of the project, the proof of concept will be put into production, integrating the solution into the customer’s IT infrastructure.

The scenario for deploying the proof of concept strongly influences implementation activities. Among other factors, two main aspects are critical to determining the complexity of this phase:

• How different the input / output format is compared to the format used in the pilot phase
• The solution’s level of autonomy with respect to client IT systems

The good news is that with Rulex, production deployment is much easier than traditional methods, thanks to its no-code approach and numerous project monitoring features.

Data-driven decision-making (DDDM) is revolutionizing almost all industries and departments. Experts agree that those who now embed DDDM into their organization will reap significant benefits and competitive advantages in the future.

This brief article looks into the topic while providing some useful tips on how to choose the DDDM project that is best for your business.

WHAT IS DATA-DRIVEN DECISION-MAKING?

Data-driven decision-making is the process of making decisions by analyzing the data at your disposal. DDDM processes may be characterized by fully automated data-driven decisions, generally driven by artificial intelligence and optimization algorithms. We are faced with this type of decision every day, for example, when we use a search engine to find something on the internet or when we look for the best way to get from A to B on Google Maps.

However, we aren’t always faced with this type of decision. When a decision involves granting a loan based on the applicant’s risk profile or launching a new product based on a market study, we cannot rely exclusively on an algorithm.

Data analysis can certainly help decision-makers take a more informed decision, but it is they who have the final say.

WHY DATA-DRIVEN DECISION-MAKING?

Data-driven decisions bring numerous practical benefits.

Frist of all, when driven by data, the decision-making process relies on concrete evidence rather than personal opinions (which can sometimes lead to long discussions, if they are not shared by others). Furthermore, data-driven decisions are transparent and reliable, and they can be continuously improved by new data.

Data-driven decisions enabled by emerging technologies are having a major impact on almost all industries (financial services, healthcare, manufacturing) and all departments (marketing, supply chain), bringing important benefits in terms of cost savings, revenue growth, and margin optimization. Many repetitive decisions can be partially or fully automated, leaving the more complex and strategic tasks to humans.

Recent studies confirm these trends. According to Gartner, more than 33% of large organizations will have analysts practicing decision intelligence by 2023, and 75% of enterprises will shift from piloting to operationalizing AI by the end of 2024.

So far so good! Fast automated decisions, lower costs, more revenue… isn’t that wonderful?

Yes, but it’s not so easy.

Many data-driven projects fail. Why?

There are various  explanations. Sometime projects fail because of a poor choice of technology – for example, a complex tool might create barriers between business and IT experts – and at other times, because of a bad choice of use-case.

Before starting a new project, it is crucial to ask ourselves some questions to avoid making the most common mistakes.

1. IDENTIFYING A CLEAR BUSINESS GOAL

The first ingredient for successful projects is identifying a clear business goal. This should be specific, realistic, with measurable objectives and a limited time frame. You can start “small”, then extend your goal after you have gained more experience.

Let’s think of two examples of well-defined business goals:

• Improving revenues by 3% in the first quarter of the year through dynamic pricing.
• Reducing the daily cost of your supply chain for transporting goods by 10%.

Conversely, increasing sales is not a good business goal since it is hard to measure and poorly defined.

2. UNDERSTANDING IF YOUR DATA FIT THE PROBLEM

Are your data representative of the problem you want to solve?

Imagine this scenario: you sell products in France, and you want to use this experience (and its data) to launch your products in Italy, assuming that European markets are similar. The Italian and French markets are probably alike (it’s up to you to decide), while applying French market data to a very different market is not a good idea.

But data representativeness is not the only issue.

One of the most common and often underestimated problems is data quality.

The expression: “Garbage in, garbage out” has become popular. If you have poor-quality data (tables full of typos, missing values, etc.), it will be hard to get something good out of them. Fortunately, there are many data quality tools, such as Rulex RDC, which achieve  and maintain data accuracy at 100%.

3. DEFINING ACTIONABLE SOLUTIONS

Can data-based predictions lead to actionable solutions, and then to achieving your business goals?  

If the answer is no, you should think again.  In the past, many AI projects were launched out of pure curiosity, without an actionable solution, and often ended up with disappointing results.

Imagine you create an AI solution that tells you the exact number of goods needed in the warehouse day to day, but you have no resources to move goods in and out. So, your information turns out to be completely useless from an operational point of view. The new solution, perhaps ordered by a curious manager, risks putting artificial intelligence in a bad light as something superfluous and time-consuming.

Carefully choosing your new project builds trust in new technologies.

4. DESIGNING A USER-FRIENDLY, INTEGRATED DATA ENVIRONMENT

It is highly advisable to design an integrated technical environment, enabling different professionals (technical experts, business analysts, and IT people) to work seamlessly without having to go through a fragmented set of solutions.

Easy-to-use, no-code tools such as Rulex Factory allow you to manage the entire data value-chain, from integration to machine learning, from optimization to visualization, from deployment in production to monitoring.

5. INVESTING IN TRAINING

Lack of competences can cause misunderstandings and delays in delivering a project.  It is therefore important to invest in training to ensure the project’s success and avoid wasting time. Full training improves communication between corporate teams, as it allows professionals from different fields to understand the project and work seamlessly on it.

Rulex Academy is a good starting point for reinforcing both technical know-how (from data preparation to machine learning, from optimization to dashboarding and maintenance in production) and business know-how (particularly in financial services and the supply chain).

Reducing churn: a real example

In the previous section we focused on theory. Now it’s time for some real examples. Let’s consider a real use-case where we are trying to reduce customer churn – a recurrent problem in many industries such as energy, retail, finance, etc.

• The business issue is very clear: the churn rate is high (12%), causing significant revenue loss compared to the previous year. It is a major problem, since, as explained in the video, attracting new customers is 6-7 times more expensive than keeping existing ones.

• The company has a good dataset which is representative of the problem. It contains customers’ personal information (customer ID, education, number of family members, etc.), customer behavior (total energy consumption, % of energy used in different time slots, number of service calls made), and pricing information.

• The number of rows is high (around 34,000), data quality is good, and no privacy issues have been detected. The company identifies two actionable solutions: offering discounts or modifying queue management. The decision depends on the output of the algorithm.

CONCLUSIONS

Data-driven decision processes can dramatically improve businesses, in all industrial sectors.

There are no more excuses for not adopting them, as they are widely accessible at different levels. However, you do need to embed DDDM into your organization consciously to avoid wasting unnecessary time and resources. Always ask the right questions (what are my business goals?; what is the quality of my data ?; and how can I change my business?).

We all know that data-driven decision-making processes can bring huge benefits to companies, such as greater confidence in taking decisions, more transparency, and a higher chance of improving in the long term.

Gartner estimates that more than 33% of large organizations will have analysts practicing decision intelligence by 2023, and 75% of enterprises will shift from piloting to operationalizing AI, driving a 5X increase in streaming data and analytics infrastructures by the end of 2024.

But at the same time, businesses often have doubts on how to deliver data-driven decision-making projects successfully.

In a previous article, we talked about how important it is to choose the right project when implementing a data-driven decision-making process. We said that businesses should think about the goals they want to achieve, consider the available data, and the actionable solutions that might be implemented thanks to their data-driven knowledge.

Let’s now assume we have identified the project we want to put in production.

HOW TO STRUCTURE DATA-DRIVEN DECISION-MAKING PROJECTS

Once the project team has been created – it should be balanced, well defined, and with all the stakeholders involved – we advise you to include 3 phases:

• Project Scope (<1 week)
• Pilot (between 4 and 6 iterations, each iteration lasts for one week)
• Production deployment (between 2 and 4 weeks)

The phases can be tailored according to your needs but, in principle, we recommend you follow them as closely as possible.

Let’s imagine your data-driven decision-making project is highly complex and requires many months of work to get to the final results.

We suggest you reduce the initial scope to give you a first deliverable in 2 or 3 months.

This has two major advantages:

• You focus your efforts on something immediately usable
• You build trust in all stakeholders, who see results from the outset.

One of the advantages of Rulex is that it is extremely flexible and interactive thanks to its drag and drop functionality and eXplainable AI. As a result, it delivers value from the very first iterations.

The graph below sums up the life cycle of every Rulex project: at an early stage, little effort is involved and the return in value is already greater than the investment.

Over time, the management effort grows slowly, while the value of the solution grows exponentially.

HOW TO DEFINE YOUR PROJECT SCOPE

The scope phase is crucial to the success of any data-driven decision-making project.

If you make a mistake in this phase, for instance, setting an incorrect goal or misunderstanding your data, the project might be seriously compromised.

Typical activities during the assessment phase are:

• Goal definition: set out the goal in detail, identifying performance measures and expected KPIs
• Data definition and audit: identify the available data, evaluating data quality and suitability for analysis
• Deployment in production: plan how the new decision-flow will be deployed in production, e.g., how it will take place within the daily work routine and how it will interact with all company processes
• Assessment: evaluate whether objectives can be met or if corrective actions should be taken

If the assessment is positive, project sponsors should sign-off so you can move to the pilot phase.

DEVELOP YOUR PILOT, FOCUSING ON VALUE CREATION

In the pilot phase, the team should provide proof of concept, and clients should see the first results. With conventional methods, customers expect these steps to take several months, requiring a large number of billable hours from data scientists, and delivering little or no directly actionable insights.

With Rulex, customers will quickly receive tangible business value in the form of understandable, actionable insights.

In this phase, typical activities are data preparation, model creation and refining, and dashboard creation.

DEPLOY IN PRODUCTION FAST AND SUCCESSFULLY

In the final stage of the project, the proof of concept will be put into production, integrating the solution into the customer’s IT infrastructure.

The scenario for deploying the proof of concept strongly influences implementation activities. Among other factors, two main aspects are critical to determining the complexity of this phase:

• How different the input / output format is compared to the format used in the pilot phase
• The solution’s level of autonomy with respect to client IT systems

The good news is that with Rulex, production deployment is much easier than traditional methods, thanks to its no-code approach and numerous project monitoring features.

CONCLUSIONS

Data-driven decision-making processes are revolutionizing the world, and they can make a huge contribution to your business.

Once you have chosen the best data-driven decision-making project for your business (see our dedicated article on the subject), execution is essential.

A few reminders for you:

• Start small to achieve more ambitious goals over time
• Try to deliver value to all your stakeholders as soon as possible
• Stay focused on your final goal

 These best practices may seem obvious, but they are crucial concepts to keep in mind to avoid making the most common mistakes.

Explainable AI it produces transparent, easily understandable models. Using a series of if-then statements, Rulex automatically produces self-explanatory logic for all decisions. Rulex rulesets make it possible to explain a decision directly to the customer or provide customer service agents with the ability to look up the reason for a decision.

Why eXplainable AI is more transparent than black box?

The problem with conventional AI is very simple: it’s unexplainable. Conventional AI relies on machine learning algorithms such as neural networks and others that have one key feature in common: they produce “black box” predictive models, meaning they’re mathematical functions that cannot be understood by people, even mathematicians.

f(x) = 0.293 tanh(0.337 x1 - 0.329 x2 + 0.251 x3 - 0.288 x4 - 0.297 x5 +
0.436 x6 + + 0.166 x7 - 0.184 x8 + 0.219 X9 ± 0.483 x10 - 0.222 x11 + 0.173 X12 
± 0.012 X13 ± + 0.352 x14 + 0.259 X15 ± 0.176 x16 + 0.345 x17 + 0.314 x18 + 0.177 
x19 - 0.329 X20 - 0.3) + - 1.934 tanh(-0.233 x0 + 0.174 x1 - 0.252 x2 - 
0.501 x3 - 0.125 x4 + 0.311 X4 - 0.573 x6 + - 0.299 x7 + 1.123 x8 + 0.318 x19 - 
1.169 x10 + 0.105 x11 - 0.429 X12 - 0.075 X13 ± 0.143 X14 + 0.146 x15 - 0.531 x16 
+ 0.077 X17 -0.133x18 0.122 xl9)

Rulex’s unique, proprietary machine learning algorithms work differently. Rulex creates predictive models in the form of first-order conditional logic rules that can be immediately understood and used by everybody. Here an example of Rulex clear box predictive model.

IF customer_province in {A, B, C, D} AND damage_class in {1} 
AND Number of days between policy start and date of accident <= 371
THEN Fraud = Yes
IF customer_province in {E, B, C, F} 
AND Customer age > 48 
AND Number of days between date of accident and complaint > 1)
THEN Fraud = Yes
IF customer_province in {G, H, I, J, K, L, M, N, B, O, P, Q, R, S})
AND Number of days between policy start and date of accident > 371
THEN Fraud = No
IF (Number of days between date of accident and policy end <= 2)
THEN Fraud = No

How eXplainable AI works?

Rulex’s core machine learning algorithm, the Logic Learning Machine (LLM), works in an entirely different way from conventional AI. Rather than producing a math function, it produces conditional logic rules that predict the best decision choice, in plain language that is immediately clear to process professionals. Rulex rules make every prediction fully self-explanatory.

And unlike decision trees and other algorithms that produce rules, Rulex rules are stateless and overlapping, meaning one rule can cover many cases, and many rules can cover a single case. This allows for fewer, simpler rules and provides broader coverage at the same time.

Rulex calculates the coverage and accuracy of each rule, making it easy to select the most effective decision rules. Also, proven heuristic human rules can be added to the predictive model, allowing a seamless blend of human and artificial intelligence. Human rules are also rated for coverage and accuracy, allowing Rulex to easily evaluate the quality of the decision rules in use and reduce false positives.

While Rulex is one of the most innovative tools for making true explainable AI, it’s not the only one. If you are curious to learn more, you can read 8 explainable AI frameworks for transparency in AI.

Where is eXplainable AI important?

Explainable AI is important to any business because it conveys trust and competence. It is particularly relevant in some sectors and applications where decisions can have a strong impact on people, such as granting a loan (read more) or making a medical prognosis (read more). According to many international privacy regulations, such as the GDPR, artificial intelligence cannot replace human decision but only support it.

Privacy is one of the areas in which eXplainable AI plays a more important role. Here are some tips for creating and maintaining processes with artificial intelligence that respect the principles of privacy.

5 Tips for Privacy Compliance

1. Identify processes in your business that use profiling and automated decisions.
2. Inventory the machine learning models currently used.
3. Assess your existing models. Are they interpretable? Can you demonstrate to an auditor that they do not discriminate?
4. Assess your current machine learning techniques. Do they produce interpretable rules?
5. Develop a strategy for meeting compliance requirements in each stage of the machine learning workflow