A Framework Implementation using Pipe and Filter Architecture


Pipes & Filters Architectural Pattern

Readers of the TopicDevelopers, Architect

Coverage Topic

  • Basic concept of pipe and filter
  • Pipe and filter architectural style
  • Basic pipe & filter example
  • When need to use
  • Pipe and filter framework
  • A tiny AI tool implementation

Let’s Drill Down the Basic Concept

A mineral water manufacturing factory have a water source. But it contains virus, bacteria, colors, salt, sand, stone, and grass. So, they need a water treatment to clean the water.

Solution: They can add few filters to clean up the dirt water through the pipe-lines. Therefore, they will be able to reserve the mineral water to the water tank.


A filter has an input pipe and an output pipe.

Pipes and Filters

A filter is connected by pipelines and the output of one filter is the input to the next filter.

Pipes and Filters Architectural Style

It has two subsystem and that is pipe and filter.

  • Filter: Filter process the data.
  • Pipe: Pipe or pipeline is a connector between two filters.

When Need to Use

If you have continuous data flow which need a chain of process.

Example using Pipe and Filter Architectural Pattern

The Problem

An organization need to verify the education history of their employees. Say, I’m the employee of that organization and I have finished my bachelors and master’s degree in computer science. The organization will consider my highest category of the degree. I mean, I have 2 degrees, bachelors and masters. The masters is the highest degree compare to bachelors.  Now they will verify the master’s degree. In this case, bachelor degree will be ignored.

Therefore, they need a tool which will find out the highest achieved degrees.  It has two parts.

  • Classify the degree Category
  • Pick up the highest achieved degree only.

Classify the Degree Category

If employees have degrees such as, Bachelors of Science and Masters of Science, then “Bachelors of Science” is in bachelors’ category and “Masters of Science” is in masters’ category.

Process Data to Filter out Highest Achieved Degree

When degrees will be categorized, then we will need to filter out the highest category and rank of that degrees.

Initial Data and Data Source

Data source – Dictionary

Taking 2 data source dictionary –

  1. Secondary words dictionary: The word itself does not have any meaning but it will help to make a sentence. So, I’m guessing 1 points for these words. The table is given bellow.
  2. Primary key words dictionary: This keyword will help to determine the degree category. I mean- if we know that, the key word is bachelors or masters or diploma then it makes a sense that we can determine the degree. So, I’m guessing 5 points for these words.

Class Diagram of Core Interface of Pipe and Filter

Implementation of Core Interface

This core-interface uses generic type. So, define your own data input models as well as sink model. We need to use it into the filter classes.

Filters Implementation

Candidate-filter class will inherit the FilterBase class and we have to implement only one single method which is called the process. Now we can implement as many filters as we need.

Passing Data to Filters Channel of the Framework

  • Initialize the objects of input data model, SingleTemplateChannel and MainChannelStaters classes.
  • Next initialize ModelForSetupFilters model class and inject the required filters into it, then send it to the singleChannel. SingleChannel will register all of the filters from   ModelForSetupFilters.
  • Setup mainChannelsStartup object like this way -channelstaters.SetupChanelList(singleChannel, educationDegreeList);
  • Finally, call the starter and get the final results from resultList objects. IList resultList = Channelstaters.ChannelCaller();

The parallel process of the channels

We can include as many as filters and channels that we need. Each of the input will pass through the Channel and each of the channel will holds a list of filters. If you have five education histories – that means you need 5 channels and it will run simultaneously for parallel process.

Example Finding Highest Achieved Degree

Input – Output

Input: Guessing a bad Input: Master of Bachelor Science 

Output: Bachelor

Filter Processing

DegreeMatchFilter: Finding the Keywords

I’m guessing a bad Input: Master of Bachelor Science 🙁

Clean-Keyword: Master Bachelor Science


From the Keyword dictionary the point for each words-

Step1: Matching Text from the Data-Source

Result depends on the keyword: Master; Bachelor; Science

    [0]: “Master of Science”

    [1]: “Master’s Degree”

    [2]: “Master of Business Administration”

    [3]: “Master”

    [4]: “Master of Arts”

    [5]: “Master’s”

    [6]: “Bachelor of Science”

    [7]: “Bachelor of Arts”

    [8]: “Bachelor’s Degree”

    [9]: “Bachelor”

    [10]: “Bachelor’s”

    [11]: “Bachelor of Business Administration”

    [12]: “Bachelor’s Degree (±16 years)”

    [13]: “Bachelor’s of Science”

    [14]: “Bachelor of Science in Nursing”

    [15]: “Bachelor of Science”

    [16]: “Bachelors of Science”

    [17]: “Master of Science”

    [18]: “Associate of Science”

    [19]: “Associate of Applied Science”

    [20]: “Masters of Science”

    [21]: “Bachelor’s of Science”

    [22]: “Bachelor of Science in Nursing”

    [23]: “associates of science”

Step-2: Count the points for each sentences

Step 3: Match and Rremove the Sentences comparing with Max point

According to the table of step 3, the maximum point is 6

So, remove all of the sentences which are less than 6.

So the output of the step 4 is:

DegreeCategoryFilter :Find out the Category

DegreeRankFilter : Find Rank point

DegreeCategoryProbabilityPartAFilter : Applying Probability


DecisionMakingInputFilter : Remove Category and Rank which is lower

Find out the Category and Rank points. According to the above table, the maximum point is 13.5. So, remove all of the sentences which are less than 13.5.

FinalDecisionMakingOutputFilter : Final Output

The max point is 13.5 and the category is Bachelor.

Therefore, the output: Bachelor of Science :==> Points [13.5]; Bachelors :=> Rank 4

So,bachelor is the highest achieved degree that needs to verify.

Category: Bachelor; Rank: 4

Demo Time!!

We never know where time taking to us. Don’t just keep your eyes on the code only. Look back your design. It shouldn’t be the victims of the worst nightmare.

Find the demo and full source codes of the project (attached).