Luca Ravazzolo · Dec 10, 2015 go to post

@Ryan I think you might be right about @Dmitry comment.

We are supposed to announce Docker containers support any minute now. We have been testing for a couple of releases. Aside the usual gotchas it's "business as usual" as long as you use supported OS container images. Right now these are RH, SUSE, CentOS. 

Your Docker engine can run on anything you like of course. That's the whole point of it and it does it fast! :)

I would recommend using Ubuntu 15.05 & above as it comes with the latest kernel and therefore does not have SHMMAX restrictions (32MB!). You'll be a happy man when you run your container and ask for more shared mem in the form of globufs :-) or just during installation time.  The alternative is to use the --privileged flag which is of course not desirable.

Best Practises? Treat it as a new platform with many things to learn about it. There is much documentation out there starting from the Docker site itself.  Of particular interest is the data persistence part, as containers are immutable by nature. How do you save data? Where do you save it to? Why should I then use containers? 

if you can get hold on our site of the Global Summit 2015 presentation titled "All about Containerization" you'll find interesting things in there. Let me know.

Containers are not the solution to all our troubles. However, they aid in many aspects of architecture design as they present operational issues much earlier in the operational process. The idea is the OP people work closely with Developers... and both find a programmatic (automation) solution to the deployment of an app.

Yes, if you thought of DevOps, you're in the right quadrant. Furthermore, think of infrastructure-as-code and even immutable-infrastructure.

HTH for now

Luca Ravazzolo · Dec 12, 2015 go to post

isn't create new post a little ambiguous? The first time I did it I thought I was going to create a "New Post"... while all I wanted to do and ended up doing was answering that thread, post. page discussion or whatever you or we want to call it... :)

thanks

Luca Ravazzolo · Dec 12, 2015 go to post

ohh darn it... I think I meant to write the whole thing above about the "Join The Conversation" button.

What does it mean I asked myself? Will I be updated of new posting in this conversation?

nope: it means answer or post in this conversation... again, I thought it was ambiguous.. not what I've seen before..

Luca Ravazzolo · Dec 13, 2015 go to post

so the right pointing green arrow? trying... :)

--

2 seconds later:

YUP :)

--

I still have to click the "Join the Conversation" button though... ;)   the description is not clear. I'd just use post or submit...

tx Timur

Luca Ravazzolo · Dec 18, 2015 go to post

this thread is an invitation to a dinner party! :-o

So many juicy themes and subjects to develop and think about... I'm VERY tight for time right now...

I'll get back to you all soon... it's all worth it ;)

later...

Luca Ravazzolo · Dec 18, 2015 go to post

@Herman:

the busybox container last time I checked was 1.1MB (yes MB) and I think the "helloworld" one was 256KB.

That is a nice portable sandbox. I agree with @Ryan: not an overkill, but just the thin veil of the sandbox :) Wed' have to re-invent a linux union FS... There is no point.  I think the trick is to start from the container you find more suitable for your specific needs like size, tools already configured, support, etc. 

As an example, I like the Tutum CentOS distribution because they provide the ssh daemon that the official CentOS image does not have. Ssh is cumbersome to provide with all the security and start-up scripts. options that we will need etc. Tutum does a great job and has been maintaining it for over a year now. Ah, they've also just been bought by Docker ;)

Bottom line: we are not in the OS business but are happy to work with these great innovations. 

You mentioned micro-service: YUP! That's where we're all going...even those with monolith...(all of us) :-D 

Micro-services deserve a thread or a GS2016 session on their own, so I won't waste this space :)

You also mention the stack... well, what about a standard VM? it's bloated; you don't care about it but you must maintain the OS; it takes ages to boot up; ditto for shutdown... isn't that bloated?

Thanks for your contribution!

Luca Ravazzolo · Dec 18, 2015 go to post

@Ryan:

Where I come from we'd say "You are putting a lot of meat on the fire" :-)

What you're referring to is the wonderful world of DevOps. The bottom line is that, yes, you can achieve all that. However, it's a new journey on a new road and like any new adventure, we'll all learn things as we do them. We can also learn from other people and architectures that we use every day ;)

One extra aspect that makes our job interesting with Caché is the fact that it's not just code, dynamically linked to an environment (container) or statically compiled (I'm thinking of C exes, NodeJS, GOlang, Java etc.) but we have to deal with a database and a container has only 3 ways to deal with persistency.

There are interesting challenges; None insurmountable :)

Again, this topic needs a session or a day at GS :)

FYI one of the best presentation -actually talking about a real working solution ref. DevOps, micro-services and immutable-infrastructure was delivered last year at the 2014 DockerCon by the then CTO of GUILT.

IMO it's worth a watch as the guy has a very good dialectic and presentation style

https://www.youtube.com/watch?v=GaHzdqFithc

We'll need a new thread on any of the above-mentioned approaches to technologies, though :-)

Thanks for posting!

Luca Ravazzolo · Dec 18, 2015 go to post

Hi Eduard,

I wanted to do that last year but I never got to start the project.

My use case was to export all messages from an Outlook format. I remember finding some script or some OSS somewhere that seemed interesting. I think that aside the 1st load the syncing is the true issue.

Do you have to go straight to the POP server?

Luca Ravazzolo · Dec 18, 2015 go to post

I agree on IMAP Eduard. I never understood why it never took off as POP did... I guess Microsoft ;-)

So, can we help each other out here?

Mine was and is just a private project.... it sounds like you have to interface with an IMAP or POP server...

Luca Ravazzolo · Dec 21, 2015 go to post

@Herman:

OK Understood. However bear in mind that with a container you don't have to worry about the OS (your cloud provider does); you just worry about "containerizing" your "service".

It sounds like you are a developer :) you need to pair up with an Op guy then :-D

You don't have to know much. Just run a Dockerfile manifesto to create your container image then ask a cloud provider to spin it up for you.  https://www.joyent.com/ only use containers these days for example... just like google.

A word of warning on data containers: they are only alive while the Docker daemon is up. I would not trust those type of containers to hold my data. Data must reside on reliable memory (NVM) and/or storage devices. Therefore, the only reasonable choice now is to mount host volumes (see "docker run -v" option). Furthermore, you can take advantage of storage level snapshots (See AWS for example), LVMs etc.

HTH and all the best with the Bento project. Let us know when available on Github.

Luca Ravazzolo · Dec 29, 2015 go to post

+1

--

Could we also add a quick filter to "show my conversations " ?

most of the time I want to just follow and answer a thread I've been on recently etc.

All should be driven by ease of use and speed

thanks

Luca Ravazzolo · Feb 1, 2016 go to post

Hi Dmitry,

This year GS is going to be different. I'm sure there will be a session on Docker containers.

in the meantime, you might want to pick a Dockerfile example and the ccontainermain exe from here:

https://github.com/zrml/ccontainermain

If you have specific questions please feel free to open up a Docker specific thread under this Cloud group.

Thanks

Luca Ravazzolo · Feb 2, 2016 go to post

I like  DO too. Great for POC and test cases as you said. Also, they have great documentation on setting things up not strictly related to their infrastructure like firewalls, Docker, Mesos etc.

Luca Ravazzolo · Feb 2, 2016 go to post

yup! They have become, over 2015, a very serious contender in the IaaS cloud arena... as long as they keep open and let people be innovative ;) they have a tendency... IMO :)

Luca Ravazzolo · Feb 2, 2016 go to post

SPOF? single point of failure? and probably other reasons? de-coupling? Monitoring of single interfaces etc.

Luca Ravazzolo · Feb 2, 2016 go to post

Hi Scott,

Why do you want to use 1 single interface for multiple providers? Don't answer "why not" :-)

SPOF can be applied to anything. If that interface is critical and it goes down, it's like if your whole production, your all cluster and solution were down. Not good. It's just like in the microservices world. No difference and dangerous. Just like in the security world: it's not IF they'll attack you, it's just a matter of WHEN. Same here. Therefore, I need to ask you a second question: What is your service strategy for availability for the Production and this BS (assuming it's critical here)?

Personally, I would de-couple and leave the responsibility of files accuracy (name, timestamp, upload time-frame vs polling etc. or whatever else) to the customers. Do you have FIFO to respect?  Other considerations? 

Ensemble is ideal for having a centralized orchestrating BP handling all the incoming requests from the business services. Your BS implementation would be simpler as you implicitly would know what customer you're reading from. Be careful also of the number of files in the reading directory. OSs have limits and issues have been witnessed before (we don't know the numbers here but splitting in various DIRs alleviate the issue mentioned).

Anyway, it might just get down to personal preferences at the end or to one particular context variable we're not aware of here.

just some thoughts... there is no right or wrong in some of these architectures... just point of views, considerations, previous experiences and best practices...

HTH

Luca Ravazzolo · Feb 5, 2016 go to post

Scott:

it depends what you mean by a deployment tool. It opens up a whole new world of automation so you'll have to start to think about versioning your artifacts etc. It depends how you want to embrace your whole provisioning and deployment and management process.

My suggestion for infrastructure provisioning & deployment: Terraform; for management Ansible: agentless and easy to learn and use.

Of course, Puppet & Chef are strong valid alternatives but you'll have to take on board other considerations...

This subject cannot be done with one post and it really depends on how much automation you want to bring to this process. I would even suggest you consider containers. 

Once automation is seriously considered there is no turning back :-)

All the best

Luca Ravazzolo · Feb 5, 2016 go to post

Thanks for sharing more info Scott. Sorry I'm late on this. Travelling...

Deployment & management: This should be totally automated. There shouldn't be the need for a GUI (it slow things down). I've given my views on another thread/post you started on this exact subject. There is much to chew on these things and you might be under time pressure, however, it's an unavoidable point (automation) if we all want to be more competitive.

Your last paragraph (human error) highlights why we need to embrace more automation. So to answer the original post question: I'd put my complexity in automation :)

I understand what you're saying BTW; I wish you well with this workflow and the whole project.

All the best

Luca Ravazzolo · Feb 15, 2016 go to post

Thank you Eduard.

So it sounds like you use other public cloud providers that offer more enterprise-level solutions like snapshots etc.

Do you use DO (DigitalOcean) only for development & testing and other providers for production?

Thank you