As i had mentioned before, I worked on the design part ths week. The clustered HPCC Systems platform has multiple components such as Thor for distributed storage, Roxie for online data processing, Sasha for backups, Dali for File Server and Authorization. These components need to interact with each other for the platform to function as expected. This is what adds to the complexity.
For designing we started with a plan to go and create separate charm for each component. But getting this done during a 12 week inteernship seemed difficult. So we moved to a different plan of creating a admin node which goes and sets up other nodes. But this also became quite complex and unnecessarily added a single point of failure. So we have now decided to create only two charms one for dali and one for other nodes. This seems like a resonable target to achieve during the course of the internship.
As for storage, I did experiment with AWS EBS for AWS Cloud and lvm for localhost. But incorporating these seems difficult to achieve in 12 weeks so unfortunately, we won’t be supporting persistent storage in the final charm.
I would be working on the dali charm for the next 2 weeks. Will be writing another post on that soon. Thanks for reading.
This is a fun week. I wrote finally started writing code. I could get the tests to work correctly. The charm test can now check if the HPCC Systems standalone charm works fine. Writing the tests was a bit difficult for me as Juju does not really provide an efficient debugging method. However, I finally, could get the tests to run.
For those who may need help debugging their charms, here are a few commands which might help you debug the charm. The commands assume you are hacking on the HPCC charm but you may use them for other by making appropriate changes.
- To see changes to the controller in real time, use
watch -c juju status --color
- To debug the hooks, you can use,
juju debug-hook hpcc-platform/
juju debug-hook hpcc-platform/ <hook name>
This would only open a session for the specified hook. If the hook name is not
specified, all hooks are debugged. Once, debugging is done, you must exit the
corresponding terminal pane for juju to continue executing.
Juju would retry running a hook automatically for a fixed number of times. If you need the hook to run manually, you can use
juju resolved hpcc-platform/1
However, his only works if the hook failed in execution.
To build, you can use the build.sh file provided.
To run the tests, goto the built charms location (by default, build/builds/hpcc-platform) and run the test using
You may need to adjust the timeouts in the tests if necessary.
- To test the stop hook, simply run
juju remove-application hpcc-platform
- To see the logs from your instance, use
juju debug-log --include hpcc-platform --level DEBUG --replay
All things said, the charm can now successfully start, stop and respond to config changes.
I will be spending the next two weeks to get the design of the clustered charm ready. More on that in the coming weeks.
This week most of my time was spent in reading the documentation and spending time exploring juju and the charm helpers framework.I also created a simple demo charm. I don’t plan on pushing it to github as it doesn’t really do much.
Ming and I during our weekly meeting have discussed the steps to be taken for the coming weeks. He has suggested I spend time on the hppc-charm repo. Hopefully, by the end of week 3, I should have upstreamed 1-2 PR’s. I plan on writing some tests for the existing charms this week. I will also work adding support for some of the items in config.yaml. ( I haven’t decided which ones but something like checksum support would be nice.)
I am keeping it short this week. Thanks for reading this.
I just finished the first week of my internship at HPCC Systems. Those who don’t know about the company, here’s a brief intro from their about page
The HPCC Systems platform is a set of easy-to-use software features, enabling developers and data scientists to process and analyze data at any scale.
I am working on making the deployment easier. I will be using Juju cluster deployment framework to achieve this. I am being mentored by Xiaoming Wang and Kevin Wang.
The internship has got off to a great start. I have been going through the docs and understanding the HPCC Systems platform in general. The platform is quite complex, with a lot of moving parts interacting amongst themselves, so I need to understand this before I really begin working. I will be writing more on this in the coming weeks.
But for today, the internship is really coming on well. LexisNexis Risk Solutions is an excellent employer and most importantly I would like to give a shout out to Xiaoming and Kevin for being wonderful mentors. I although am new here and don’t know much but these guys have been at the forefront to help me out whenever I need. So yes, thanks a lot Ming and Kevin. Looking forward to working with you.
That’s it for today. Thanks for reading this.
Finally my first blog post!! Thanks to Andrew Fong for this awesone hydeout theme.