[chef] Re: Re: Re: Re: [Chef] Communcation: MyApp-ChefClient-ChefSrver-ChefClients

[Leonid Medvedev < >]

Hi all.

Sorry for the delay - investigating.

After investigation the really interesting suggestions I see two cases which will be investigated deeply. I will describe them and ask your opinions.

Case 1:

 - MyApp. will have to protocols: REST API (for receiving responses) and Webhooks (PUSH notification for sending requests) for integration with Configuration Managers (CM) Chef/Puppet/....

Briefly the sequence of integration with CM:

1. MyApp. (from host 1) prepares request - create Push notification.

2. CM Server receives Push notification.

3. CM Server performs the received request.

4. CM Server prepares response for MyApp - create PUSH (REST API).

5. MyApp receives REST API response from CM Server.

Only one big question: we should develop some "bridge" which will receive PUSH Notification and re-transmit them to CM Server. After that this "bridge" has to receive data from CM Server and re-transmit as REST API for MyApp.

Do you have ideas about "case 1"?

Case 2:

 - MCollective. This is smart solution but it is only for Chef/Puppet/Salt. As I understand MCollective can help integrate (re-transmit data) MyApp with CM (Chef/Puppet/Salt).

Thanks for cooperation.

Leonid

2015-02-05 6:12 GMT+02:00 Ranjib Dey < " target="_blank"> >:

1. pushy is chef push jobs :-) . if your app is written in ruby you can use the chef library itself to access chef server, it provides a CRUD api for all domain objects(node, role, environment etc), a search query api etc. If you are writing applications in go the go-chef can be used to do similar actions. Noah one of our awesome community member wrote python binding for chef, jcloud has a java binding also  (and i think an android app is also there). All of these library allows you to do the communication with chef server rest API  (i.e. auth headers, http, json handling etc). Depending upon what stack you are choosing to your custom SDK you can choose one of these, to those crud stuff or search query.

2. You can extend chef-client in many ways, some popular ones are:

a) cookbook -recipes

b) lwrp/hwrp - resource providers

a) report and exception handlers

b) event handlers - execute your custom logic during certain time in chef run life cycle. 

c) custom knife plugins - build custom CLI tools that involves talking with chef server

Recipes and cookbooks are actually an extension of chef (since they are eval-ed).  

The example you have given can be done by standard resource provider itself , the database cookbook offers user and db as resources. 

If the intention of db alteration means a configuration, and you want to keep it consistent across your infra you can offer them as resource/providers. Depending upon what task you want to do you have to choose the extension mechanism. Like earlier someone was asking about displaying version specific recipe usage across the nodes. Knife will be perfect for the display/cli part, and an event handler can gather the custom data and save back as a node attribute

3) chef client can only post back data about itself. So it can save any update about itself. I wont call this as event transmission though. Neither i am aware of anyway to retransmit chef-server changes via chef-client. 

Note: all these experience is based on opensource chef server, push jobs and other chef-server add-on might be providing more stuff, but i have never used them.

Most of the earlier orchestration frameworks i have mentioned are opensource :-), as long as they solve new challenges, that can not be easily done by existing solutions, i think its fine, as otherwise you'll end up rolling your own any way.

[1]https://github.com/go-chef/

[2]https://github.com/coderanger/pychef

[3]https://github.com/jclouds/jclouds/blob/d3c1e2eab4bd71ecdf2ebfcd73e5d28c835398cd/apis/chef/src/main/java/org/jclouds/chef/ChefApi.java

On Wed, Feb 4, 2015 at 12:06 PM, Leonid Medvedev < " target="_blank"> > wrote:

Hello Ranjib Dey :)

Thanks a lot. Sorry for the delay with answer.

Unfortunately we are not familiar with these frameworks: mco, pushy, ansible, blender or rundeck. But we will investigate them; you understand it is not easy to use external additional products.

But I have several additional questions:

1. Is it possible to use "Chef Push Jobs" (https://docs.chef.io/push_jobs.html). I would like to install this service on the same host with MyApp. And the question for these case - you wrote: "Your app can talk to chef server using the ruby or java or go api", please clarify or send the link with explanation.

2. Next question. Is it possible to extend Chef Client - for example it should open DB and read/write some fields in a table. Please also clarify.

3. Last question. Can we use Chef client as retransmitter of messages to Chef Server?

Thanks a lot again and again :).

Leonid.

2015-02-03 23:40 GMT+02:00 Ranjib Dey < " target="_blank"> >:

Do It :-)

Its possible

"The Chef Client has to get trigger from MySDK (it can be alarm or field from the database) and sends the trigger to Chef Server (second host)."

 - MySDK has to use mco, or pushy or ansible or blender or rundeck like tool to do this (orchestration engine).

Chef Server does not support plugging in custom modules (last time i checked there were talks about http middlewares, but nothing concrete),

I'll suggest using chef server as artifact store (cookbooks, roles , databags etc) and metadata source (chef search, nodes etc), but not as a messaging system (i.e. chef server triggering client runs), its a different service , you can consolidate them in one box but still they are different.

I have used MCollective[1] to do the same, now im using Serf (fleet wide triggers, and all other automated triggers) and SSH agent (for on demand runs) via blender[2]. You can do same with pushy, but i dont know the full feature set there.

In this scenario, you'll  use chef event handlers to drop serf events or mco calls to indicate state change (like run failed or node converged or a particular file has changed), these event can have payloads. Your app should subscribe these events (like serf handler, for mco u'll use exchanges, topics etc AMQP primitives). Your app can talk to chef server using the ruby or java or go api (whatever language you choose), chef' client's REST abstraction is all you need to talk to chef server.

It will easier to discuss an actual use case (e.g vm provisioning via web service), then we can break down the use case into tasks, and then provide exact examples and compare them side by side :-)

cheers

ranjib

[1]MCollective: http://puppetlabs.com/mcollective

[2]Blender: https://github.com/PagerDuty/blender

[3]Serf :https://serfdom.io/

On Tue, Feb 3, 2015 at 9:07 AM, Leonid Medvedev < " target="_blank"> > wrote:

Dear all.

I have one interest question. Please answer is it possible or not? Please advise what is the best choice from your point of view.

The description in details (the picture is in attachment):

1. Hardware configuration:

  - The first host has: MySDK and Chef Client;

  - Second host has Chef Server;

  - other Chef Clients installed on a lot of hosts (for example 100-1000 times).

2. MySDK installed on the same host with Chef Client.

3. The Chef Client has to get trigger from MySDK (it can be alarm or field from the database) and sends the trigger to Chef Server (second host).

4. The Chef Server sends a request to other Chef Clients which have to perform cookbooks according received trigger.

5. Chef Clients perform cookbooks and notify Chef Server about results.

6. Chef Server sends this result to the Chef Client (first host).

7. Chef Client informs MySDK (they are on first host) about results of the trigger.

Thanks&Regards

Leonid