Infrastructure is key in company growth
Chances are you arrived here because you need advice or tips on your company's infrastructure. Your booming company is making the transition because it has out-grown old procedures and maybe the company has finally decided to move into big-data and simply doesn't know how to start. You are not the first and will certainly not be the last. It is usually best to take a step back and realize how you got to this situation.
There is a point in which companies realize they need to start collecting larger amounts of data to better assist their customers. This generally occurs when a company is transitioning from a medium to a large size one. There is no set revenue or number of employees to discern this point, however, the company has just had its head to the ground trying to increase profit margins as much as possible and after many years of success there is a jolt.
Management will try to take a look at the current processes and realize the company grew too quickly for its own good. They will notice there are now horribly inefficient practices to simply keep up with the amount of work. Also the company has become very specialized and segmented that there is no proper way to reconnect them for analytics.
Health First, the hospital, and Health Plans, the insurance company, were both forced forced to purchase from vendors whenever a new necessity arrived. Then when this jolt occurred, a large scale procedure-review made them question why they were paying for the profit margins of around 15 other companies. Along with the realization that there could be no analytics or connection of the data between the insurance and the hospital data. Metaphorical band-aids were placed on the problem as it continued to grow and only get worse. We currently assist these two companies to develop their eventual goal of predictive analytics and big data. It takes time to repair this much damage.
The one thing to note here though: many large companies seem to think they can do this on their own. When hiring a full team of lean professionals at 150k-200k each isn't an option, companies tend to solve this problem by dedicating a team to simply 'make it happen'. We have seen far too many companies fall into this pit and contact us after they have seen an entire year of attempts with literally nothing to show. Nearly every consult we attend has the company needing to revamp it's entire process. There is a finesse to developing the infrastructure - we've been through it multiple times. If your team is not currently discussing one of the following you need to contact us now...
- NoSQL database clusters
- Parallel processing framework vs Serverless processing
- Cloud-like reliability
Please do not think that you can avoid key infrastructure upgrades to achieve specific big data benchmarks. See who else agrees with us: IBM, Google, Amazon, Walmart, literally every other global company.
You Need Enterprise-level Programs
Stop using Excel and Access to hold your data!
One of the keenest signs of an lacking infrastructure is when a company is still using excel sheets to record everything. We should all be able to understand the large problems here. How many times have you been looking for the 'correct' data when there are dozens of files all with different dates? How many hours do your employees spend fixing their personal excel sheets?
Excel files are easy to keep track of when there are only small amounts of people using them. These are usually one of the first methods companies will use to keep records of their customers, revenue, and expenditures. I have seen far too many companies drag this out as long as possible even to where an entire hospital is still using excel sheets to keep track of every patient's pharmacy records. You never know if the data is actually correct, if its been modified, or who all can see it.
Access databases are only one small step above using excel sheets. These pop up in companies when one moderately tech savvy person becomes employed. Many problems still exist here and will not be able to progress for large companies. Access at least removes the main concern of knowing if your data is the 'source of truth' or not.
However, Access files are still simply files sitting on a hard drive, they can be deleted if security measures aren't taken, there are database size limitations, and even processing limitations (see here). Access has some heavy limitations. When companies have unlimited usage of Access, these tend to pop up everywhere and for every thing. Eventually you come to the same issue as excel sheets and still do not know which data is correct or if its still relevant. Who knows how many hours as lost...
One company we are currently assisting is currently underway of removing the 2,200 Access databases on the network. These will be placed into centralized data sources so there will no longer be any possible confusion.
Instead try enterprise grade databases!
Everyone seems to think there is some catch to upgrading to real enterprise-level database programs. This could not be further from the truth.
The only catch is that SQL Server and similar programs are harder to configure at first - that is what we are here for! You do not need to hire full-time specialized Database Admins to take advantage of these necessary components to grow your company correctly.
Unlike Access, SQL Server Express is actually free! Many companies will not need to proceed beyond the express version.
Enterprise level databases allow the configuration of network databases instead of files. The key here is that your excel/access file can not be erased on accident. Anyone with the access may connect and query this data as they need unlike how Access may block and restrict the file because it is currently in use.
Enterprise databases do not have limits to their growth. Access has a single processing thread and a maximum of 2 GB per database. Enterprise solutions have no such thing. When you are trying to query and process data from thousands or even millions of records, a single processing thread is nothing more than a bottleneck. Not to mention, how will any company respond to suddenly not being able to modify their customer data? Make the first move before it's too late!
- One true source of truth
- Can actually be free!
- Easily Upgradeable
- Network Node availability
- Better security
- Save time and money!
Connect Your Data & Correctly
Lawsuits & Audits
Audits happen to nearly every company at some point. It is far too easy to get hit with numerous fines for anyone one of numerous reasons. The last thing a company wants is the government breathing down their neck and while they still need to work with their customers. It places too much stress on everyone and never ends well.
In health care, lawsuits are unfortunately very common. Hospitals can sometimes lose lawsuits if they can not prove the patient's treatment between all doctors in the network. One lost record, one error, one typo. That is all it takes to lose a court case.
As described previously, Health First grew too quickly - way too quickly. When another necessity of the company was discovered, they went and purchased the first thing they could find. This turned into 4 distinct areas of patient data. Let's not forget all of the employee data that was needed to be able to track who changed what data and when.
These 4 systems became the backbone of the company and could not be replaced. Thousands of hours were allocated to connect these systems as best as possible but it was never enough because they did not take time to plan ahead. Our team was placed right in the middle of these modifications and we finally made tremendous progress.
Standalone systems tend to restrict access in and out of the databases. This requires back-end modification to allow customer id's to pass between systems. You are not allowed to simply use customer SSN's to perform this - you will eventually be audited and severely fined! Don't do this!
Once you have these customer id's in place, the fun begins. The easiest method to fix data connectability is the usage of API's. You can use an entire IT team to build custom API's or use products like MuleSoft, again, this can be free!
API's & Mulesoft
How do you think so many websites and businesses are able to "Find the best rate"? They do this by accessing other company's API's! An API is nothing more than a way to return formatted data. There is no limit to what you can send & everything is as secure as you want it.
- Can be free!
- By far, easiest way to link systems
- Software already exists to save you time and money!
Best and Cheapest Processing
Every company started out as one or a few computers that had their own processor. It worked for the time. As time continued and the company grew, the necessary processing skyrocketed. There is no way a single standard processor could handle the necessary workload for a bank, a hospital, or any big data gathering. From here, you have two options:
Go buy multiple thousand dollar processors and hook them all into connected mother boards. You will easily spend $4,000 each if you need top of the line equipment. These can easily break at any moment but there is no middle-ground on this price. You will need to purchase the item again.
There are other solutions to this problem. The easiest is to create parallel processing clusters. You can have as many as you need yet provide the same power at a fraction of the cost.
Every company should want to get the most processing for their cheapest price. With parallel processing clusters you are able to easily add more processors when your company needs. If one individual part fails, it will cost you 80$, not $4,000.
Every one of our team members has configured these clusters for companies in their previous experience.
Even if you choose to get the expensive processors with 20 cores, they are not always able to efficiently use their processing. There will be slow periods of processing and at this point your thousands of dollars are just sitting idle providing zero value.
You are even able to hook these super processors into the clusters as well. So even if there are slow processing periods in some business aspects, the unused processors are able to take some of the work-load and help you get your money's worth.
Apache Storm Clusters
- Used by numerous large companies
- Easiest and quickest method to install