Microservices: Shared libraries or big services

After some fun with microservices a fork in the road has appeared.

Say we're developing an app (just like I am):

The web-part doesn't exist yet, it will be the web GUI (Spring boot MVC).

We have two small services:

• UserRegistration
• UserInfo

Since they use the same data I choose to group them with the same database.

So when we register a new user, the info will be sent to the UserRegistration-service by AMQP-message via RabbitMQ.

The UserRegistration-service puts the data in the database. To control the structure we make a POJO (JPA or other, depends on the database eg. MongoDB).

And there is the problem. The structure (the POJO) is in the UserRegistration-service project!

So when the UserInfo-service retrieves the data you probably want the same structure as you put it in.

There is probably more solutions but...

I see two obvious roads to go about:

1. Make the services bigger and have all the user-related stuff in one big service
2. Have a shared library with the code that needs to be shared

Personally I think option 1 is too risky. The service might become a giant service so the "micro"-part might disappear. 

So how do we do option 2? 

The code could either be a separate project and use some form of distributer to a artifact repository or try to make it part of one of the services (probably the one who puts it in the database) and try to package that part of the code some how.

This adds another level to the complexity. The need to share libraries, that the services can't be 100% independent.

This is a big part of the microservice architecture, it is somewhat complex. There is a lot of benefits to reap but you can probably solve this much easier in a monolith. 

On the other hand there is a lot of benefits if you can get the microservices right:

• Fast development
• Independent deployment of services
• Scaling
• Services can be in different programming languages

So now I think I will deploy a Sonatype Nexus repository as a Docker-instance and share the User-structure.

To be continued...

Quick update:
After reading a bit about others experience concerning microservices another problem came to mind.
https://rclayton.silvrback.com/failing-at-microservices
If you have shared code between services, especially Models (entity structures) this is most likely written in a specific programming language. This takes away one of the aspects of microservices.

So how do we solve this? This will take some thinking (I would like a good place to discuss these problems). My first thought is to find some JSON-structure due to the AMQP and send all messages by JSON-format (formatted as Byte arrays). Hopefully this makes it possible to use the message in another language.

And Happy New Year!

Microservices continued: The messaging begins

So I've started with a fun project: http://java-viking.blogspot.com/2014/12/mircoservices-with-java-spring-boot-and.html

So the first piece of the puzzle was to have individual service running. This is easily solved with Spring Boot. Since most services doesn't even be need to run as Web services (run in a servlet-container) it can be as small as a few mb!

So I wanted all the communication between the services to be solved by messaging.
The "normal" way is point-to-point. The problem is that you are bound by the API, sure you can do good coding and use a interface so you actually can switch the implementation. But somewhere the implementation is bound. When you need to do a swap, that redeployment will disturb the things that uses it.

Can you do a redeploy without disturbing the system?

Answer: Not sure, but I will try!

My thought is this: If you have services that uses messages queues then you can theoretically have several instances of that service handling messages in the same queue (not the same messages but the same type of messages).

If that works then you could have an instance of the old version and an instance of a new version (where a bug is fixed) and then remove the old instance and work would probably not have been interrupted.

This is also the solution for scaling that I'm testing. Can you have several instances of the same service?
So far so good, my fear is when users and session come and messes things upp.

AMQP

So instead of JMS (Java Message System) I'm using AMQP (Advanced Message Queuing Protocol).

One of the reasons is that is has clients for several languages so you can have some services with another language.

"AMQP mandates the behavior of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous attempts to standardizemiddleware have happened at the API level (e.g. JMS) and thus did not ensure interoperability.^[2] Unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language." (Wikipedia)

Spring, of course, has a project for this: Spring AMQP. I'm using Spring Boot Starter AMQP (not fully sure yet what the "starter" is about)

Status:

So where am I right now (December 23)?

I've done a service which listens to a queue on RabbitMQ (I named the queue and RabbitMQ is running in docker, use the official dockerfile/rabbitmq)

This service will handle every message in that queue, it will print what the message is and change the message.

If the service that sends the message demands that it wants a response, that service will get the changed message.

The other service just sends 10 messages (with Thread.sleep(1000) between each message) to the queue. This now works like a charm!

But I had some big problems with this! First off, I'm new to spring boot and messaging.

So I could quickly make these services with examples from the web.

Problems:

The problem was that this only worked with one instance at a time! No bueno!

I started the MessageHandler-service, no problem.

I started message-sending-service instance 1, no problem.

I started message-sending-service instance 2, the messages got jacked by instance 1!

And instance 1 saw it as the message was sent after the timeout (no kidding! since it actually already got a response before!)

The problem was that the MessageListenerContainer that was used was still running after the response-demand was fulfilled. 

If I stated instance 1 and 2 and then started the message-handler (love queues!) they got there responses correct.

So how do I fix this? (many hours of fun)

First came the o-so-fun task of actually finding out what it's called. (send-respond/receive-respond and more)

It seems that most people use either a separate queue for each client or each request!

This seemed like a performance hog.

But in RabbitMQ 3.4.0 there is a function for a pseudo-queue for just these occasions.

It's called Direct-to. We let the code and RabbitMQ handle the response (so that the right message comes to the right sender).

The Docker-RabbitMQ was version 3.4.2 lucky!

Next problem: this is supported in Spring AMQP 1.4.1.

What version does Spring Boot 1.2.0.RELEASE have of AMQP? 1.4.0....     O joy...

I tried that version and it didn't work. I'm not sure if Spring Boot allows me to switch a version of one of its sub-projects. I didn't get it to work, but it might just be my incompetence .

So I took my chance with version 1.2.1.BUILD-SNAPSHOT! 

And hey what do you know, it worked! I deleted the response-listener completely in the message-sender, the Direct-to function handles it for ous.

Minor setback:

The "pseudo-queue" for the Direct-to function https://www.rabbitmq.com/direct-reply-to.html#use

Have an official queue name " amq.rabbitmq.reply-to", it needs to be exactly this!

I stumbled upon this in a StackOverflow answer. The Spring Documentation actually had a typo of this: http://docs.spring.io/spring-amqp/reference/html/amqp.html#direct-reply-to

under section 3.7.1 it said " amqp.rabbitmq.reply-to" that "p" should not be there!

And I understand how easy of a mistake this is, I thought it was the other way around, why would it be "amq" instead of "amqp"?

But no problem, I submitted the typo to the Spring Jira issue site (issue AMQP-458) and the awesome Gary Russell has already fixed it, it'll be in the 1.4.2 version: https://github.com/spring-projects/spring-amqp/commit/fca82ce3d54bb2024efa6275a2e661cd20829de6

You got to love Open Source!

I'm happy to help fix this, cause you don't get any fun errors if you do this typo.

You just don't get any response. The message gets handled but that's it. Now the message-senders is just waiting for the postman that never will come.

So I hope this will help someone, I will keep you posted with my findings and results. In the end I will probably give some code, just now it's not so pretty ^^ except the gradle, that is pretty!

If your trying to replicate my idea and get stuck, don't hesitate to comment and I will try to help you as best as I can.

PS: These services will have there own Docker-containers and be run from there. They don't need to know about each other or there respective IP-numbers, you gotta love Messaging systems.

So code away!

Mircoservices with Java, Spring Boot and Docker

Intro to microservices

One of the current buzz-words flying around is "Microservices".

Most big java apps (web-based at least) are packaged as a big WAR-file. The project is one big app, a monolith. This comes with some problems concerning development.

First the obvious, if you change anything you need to redeploy the whole thing. Even the smallest thing.

Second is scaling. If you need to scale up then you scale the whole application.


So what is microservices? Instead of having a monolith you make services as separate projects.
A service would have just one area of responsibility ex. Login-service.

Pros:
Fast development and deployments
Easier to write tests (small project > don't disturb other tests)
Can redeploy that service without redeploying everything
Scaling, just start another instance of the service (needs a good framework for this!)

Problems

So is it a hail mary-technique?

As always, this technique/architecture has its own flaws and problems.

The main con is the complexity of service-handling. The services needs some way of connecting to each other. And "hard-coding" the connection gives us a BIG risk. Because if you change one service, all the services that uses the API of that service might crash in runtime, fun to be that admin.

And how do we know that a service is down?
Say the login-service goes down, the positive is that the app might still work for the people that is already logged in. But no new logins can occur until the service is up again.
With a good health monitoring system ex. Nagios it can check if the service is down, some can even fix the problem!

Solutions

So how do we solve these problems?

First the decoupling from the API. If the services are decouple and doesn't use the API one-to-one we'll escape the hellish nightmare of the total system crash (in this case, might be others).

So we can use some form of integration system like an ESB (Enterprise Service Buss) or just a messaging system (ex. RabbitMQ or ActiveMQ).
This will send the message in to a queue, the service will then grab the message and handle it (complexity gets higher).


Second

A awesome system would be some form of clustering over different locations, load balancing, service discovery and health monitoring.

Some frameworks can actually call the "mother ship" and say "hey I'm bla bla service and I'm ready to Rock! I'm at this location".
The service would have an adjacent health monitoring script.

Is there an actual solution for ALL of this? Answer: Not that I know, but there is some puzzle-pieces.

My Experiment (the plan!)

I'm going to try to make a simple project using microservices.

I'm going to use Spring Boot for the service. Spring Boot is a small application which has a jetty/tomcat server in it and it can either be run as a WAR or by itself.
I'm going to run it by itself. This becomes a app with 12-20 mb per service.
Gradle will be used for the build and dependency scripts.

After the service-app is done, I will contain it in a simple Docker image that is built on start.

Ain't it a problem to start all these docker containers (I will have 3-5 services)?
No, I will use fig (http://www.fig.sh/index.html) to start all the services at once, and to document all the start-commands, linkages, port exposures and more. To sum it: A docker kick-starter.


Ok so Java: check
Containers: check


So regarding the health monitoring. I will try out Consul (https://consul.io/intro/index.html).
I hope I can rock it good with docker and use both the service discovery and the health monitoring.

Onwards

So right now I have made a test-service (hello world in spring boot) and made a docker image of it and ran it successfully.

I will keep you posted of my results, hopefully I can put it all on github when it's done so you can get all the source code.

Another hope I have is that Docker Swarm takes of and either becomes powerful enough on its own or that some awesome frameworks gets created that uses it.

Docker and Mule

So I've been looking in to Mule ESB (Enterprise Service Bus) for my new job.

After testing it, it seemed like a lot of fun. Another of my projects was to learn Docker (scorching hot right now).

So I thought "Why not combine them?", little did I know that I was in for a world of Internet-searching, documentation exploring and more.

So first some more Info on Mule and Docker:

Mule ESB

http://www.mulesoft.com/

Is kind of an adapter between services. It's message-based and has a lot of functionality already done.

Ex: Say you have a Java site that needs paypal. Mule can sit between, take the request by JMS, HTTP and more and pipe it to paypal and then back again. Maybe you want a facebook post for every 1000 visitor? No problem.

Why do this when you can code it your self? One of the reasons is that you can use much of the code again, you can separate some of the work to this Mule server or you can do a microservice style and have Mule be the glue between them all.

Docker (Docker.io)

https://www.docker.com/

Docker is a way to make Linux Containers (LXC) manageable and share-able. It's a technology for making small VM's where you can isolate you app.

Why? Say that your a developer. Your app works in your developer environment. But that's not always the case for the production environment. 

What if you could have a environment under development and just ship the whole environment? That's what you do with Docker.

It's also a good subject if you want have microservice's, every service can have its own Docker Container. 

And it's scalable, you can have servers with CoreOS (made for running in HUGE clusters, we're talking Google-size) and run your Docker Container.

Cause that's one of the best perks, your Docker Container runs ANYWHERE where there is a Docker Engine. You can develop under Windows and Mac with boot2docker via Virtualbox and then Deploy to a real Linux Server running Docker Engine.

 And people share many of their Docker Containers via buildfiles (DockerFile). You can have your own private registry of Images or use the Central DockerHub.

My Problem

So there is a couple of people that have done Docker Containers for Mule Server.

And I could not get this to work!

I was a rookie at Docker so I blamed my lack of knowledge.

I followed the guide and it would not work.

The regular Mule Server and my test app work perfectly.

The Docker version blocked the right ports, but it would not give me a response.

I scratched my head...

Spinning up a Wildfly server in docker was 2 commands and it worked perfectly. 

So I compared the DockerFiles, but nothing looked weird. Was there something special with Mule?

But why? The creator seemed to get it to work without problem.

After that I found a solution!... Or so I thought. I was able to publish the port so I could run and get a response...from localhost but not from a remote computer.

But wildfly worked with the remote computer, what was the difference???

Frankly I thought I was missing a puzzle piece with Docker, like firewall, config or something.

After 2 weeks of leaving and coming back to it (Googled some on breaks at works, coming up with new ideas what it could be) I finally found the answer.

After being really frustrated I started to really look at the test app and compare it to the Docker Container Creators test app.

My app said that localhost should run the http start-point, the other app said 0.0.0.0....

OF COURSE! So I was looking at the wrong place!

I thought I made the simplest test app ever, a little hello world, but I succeeded in not doing it correctly. 

After changing it to 0.0.0.0 it worked flawlessly. So simple that it's extra hard to find.

But on the up side I learned A LOT about Docker and how to use it.

So I can recommend Docker and Mule, Docker is not as hard as some think, just know what your putting in it before blaming Docker.

Code ahoy!

Spring MVC Step 1

So I've been digging around Spring. It's a very versatile framework. So you might need to research which parts you actually need.

I'm into Java EE Web so I want the core-part and the web-mvc (yes they actually have a subproject tailored to MVC on the web).

So I went and found the easiest tutorial I could find. Some tutorials are great but they give you like 50 classes from the start. If you'r trying to learn Spring then you might want to know if all the classes are needed or not.

But I found one that was small with good explanations (http://examples.javacodegeeks.com/enterprise-java/spring/mvc/spring-mvc-hello-world-example/)


So I will first show my setup (I use Gradle instead of Maven like the tutorial uses).

Folder-Structure

Project
--src
----main
----test
--build.gradle

The src just hade a package structure for one class, the HelloWorldController.java
Right now the test-folder is empty, it will be used for future junit-tests.

build.gradle:

The rest is identical to the tutorial. If you want to run it in a Wildfly-server (the tutorial uses tomcat), you just run "gradle assemble" and either copy the war to the deployment-folder or you can use the web-managment and deploy it there.

So now some explanation of how it all binds together:

DispatcherServlet

The most central part is the DispatcherServlet. This is a servlet that Spring have created to mediate all the traffic. The http-request come to dispatcher and it then uses the RequestMappings from the controller-classes to retrieve the right view and model.

It reminds me a bit of the FacesServlet that all the facelets with JSF goes through.

So in web.xml you define the DispatcherSevlet just like you would do any (before the Annotation-trend started) and map it to the "/" url-pattern.

When you define the DispatcherServlet you may pick a name for it. It will then look for this name as a xml file: "name-servlet.xml" in WEB-INF.

So what would we set up in this xml? How the dispatcher will resolve the connections.
You can define all the components one by one, do a component-scan or a component-scan via Annotations (not in the xml).

In the tutorial (which I also use) they defined a bean, in this cause a bean of "org.springframework.web.servlet.view.InternalResourceViewResolver". This will be used to resolve the views.

In it we set some properties:
prefix: the folder where the views are.
suffix: .jsp

This makes it easy for the classes. They can return a String "hello" and it can be resolved to /WEB-INF/views/hello.jsp". This makes the Controller classes more readable.

Then we need a Controller to actually do anything. I put a "index.jsp" that just says welcome. It shows when the "/" url is requested. My web-app folder structure looks like:

Controller

A controller is a class with @Controller annotation. It is likely that you want the @RequestMapping("/hello") also. 

You can then do a method that returns a String. This method should have the @RequestMapping(method = RequestMethod.GET) annotation. This only registers so that when a request is used via GET, this method is returned. 

This means that you can have seperate methods for the others ex. POST

You can now return "hello" or "helloWorld" depending on what you named your jsp-file.

Conclusion:

We need:
A web.xml which configs the DispatcherServlet
A dispatcher-servlet.xml which configs the view-resolver.
A Controller

The dispatcher will register all the mappings from the controllers and direct the traffic.
The controller will return the name of the view to the view-resolver which puts prefix and suffix on it.

If you put your views in the WEB-INF users can't directly request them, they have to use the url-mappings of the controllers, which is a very good thing.

(EDIT: and after the ViewResolver sends it back to the dispatcher the dispatcher sends the html to the user that requested it)


So that's the first step in Spring Web MVC. I find Spring very interesting and will continue to learn and explore. I will continue to use Wildfly and Gradle. One of the things I want to do is an automatic deployment without a IDE. You can do this with Cargo-plugin but I want to find a different way.

Happy Coding!

Early Spring

I'm almost done with my first real Java EE project (JSF, JPA, Wildfly) and its currently running on a raspberry pi.

My next project will be learning the Spring Framework.
I will also try to do some post about how to start with Spring.

Initial impression:

My first impression of Spring is that its a well made framework that fills every part of your development (specific test-suite and more). The different modules of the framework really gives you a good feeling of security for functions.

But there is a big step for a newbie to just jump into Spring!

First problem: Overview

What does what and how "do they want you to code". With pure Java EE you can do as you want,

but in Spring there is a certain way "they" want you to code. I'm not saying it's bad. But there is a problem of learning it.

Second problem: Too big tutorials

Many of the tutorials start to big with 20+ ready to use classes. And the tutorial uses these classes. This makes you feel like you jump into the middle of things and not understand the whole process.

Third problem: Spring Boot

Lovely little thing that really can kickstart your project, but it doesn't help you in understanding much.

Fourth problem: Name usage

Some of the names conflicts with pure Java EE7 names. A Spring Bean is configured differently then a regular Bean that can be Dependency Injected.

Dependency Injection in Spring is different then Pure Java EE7. EE7 has @Inject while Spring uses @Autowired that usally is a Interface that a config-class or a xml has mapped to a implementation class.

So that's some of the hurdles a rookie will deal with.

On of the links I've found useful (haven't had time to read to much yet) is: springbyexample.org/examples/core-concepts.htm

This site ACTUALLY explains the core concepts.

So when I have some more information about the core concepts that are essential I will make a post about it.

Happy Coding!

TDD - Test Driven Development

Intro:

One method for coding stable code is TDD.

The basis of TDD is to write the test first, test it (and get an obvious fail) and then write the implementation.

Why would you do it this way?

First:
Because you implement the classes as a request-contract.

Second:
You have value full tests to-go

Third:
With these tests, if you refactor code hopefully your test will alert you for any breakage

My Story:

I started on a project (Java EE 7 with JSF) and wanted to practice junit tests. But for Injected beans and Jsf-contexts, the story got tricky.

Now the Great test framework Arquillian has all the answers for that, BUT some of the things you need is only in alpha state. I struggled with it a lot. So much I actually choose not to do any tests...

BIG MISTAKE!

So when I tried to code a function for deleting a certain thing from the database (in the middle of relations) i got some problems. Every time I changed the code, I had to log in, navigate to the right site, try the function and then try to understand the giant EE stacktrace.

So I did a GIANT Refactoring (actually starting a new project and just copying in the entities).

I tried for 2 days to implement all of Arquillians might for testing actual websites in the server (Graphene, Selenium, Drone). But in the end it didn't work.

But I got the junit tests (run by Arquillian) to work (as injected beans).
So now I could actually test the beans function without all the login and stuff.

And when it's all implemented I can actually do manual Selenium tests to see that it actually works in the website.

A bonus thing is that the test wipes and then populates the database with test-data.
So if I have to drop the database (ex if I change the persistance classes) I don't have to create all the test data manually.

Summary:

So what can we (and especially I) learn from this?

Even if you don't do the tests first like TDD, have a test framework! It saves so much time.

Code ahoy!

ManagedScheduledExecutorService and Wildfly

So I needed a task to be run once every day. I needed to check if a user had 7 days left of the membership (and mail her/him a notice) and I needed to remove the users and the connected stuff if a members membership had expired.

I thought I would use a cron-job (linux for the win!). But I had made these nice functions with JPQL that retrieved all the users that had the requirements.
I actually stumbled upon the solution quite quickly but boy did it take some annoyance to get it right.

The answer was ManagedScheduledExecutorService. That's a mouthful.
This is a Java EE7 specification so the servers need to implement it to call them selves EE7 Application servers (the web-profile-only servers doesn't count).

With this you can schedule a task (any POJO that implements Runnable which is Java 101).
You can set it to run once, with fixed intervals or fixed delays or you can implement a Trigger (a interface) that sets of at certain times or conditions.

I went for the intervals and Wildfly. Now with Wildfly you can either use the standard config for ManagedScheduledExecutorService or you can config you own. I choose to config my own because the standard one had several threads (4 I think) and I only needed one.

This was done in the config of the server (standalone.xml/domain.xml).
You can also config the threadfactory there.


<managed-scheduled-executor-service name="removerScheduledExectutorService" jndi-name="java:jboss/ee/concurrency/scheduler/RemoverScheduledExectutorService" hung-task-threshold="50000" core-threads="1" keepalive-time="5000" reject-policy="RETRY_ABORT"/>


With the long jndi It actually shows up in the Wildfly server managment interface,

Problems

So I had two problems:

1: My intervals wasn't working at all... For some reason it started 3 threads that ran 3 seconds after each other (I had 15 seconds intervals for testing). And then all 3 threads had 15 second intervals between there own executions. I made it print out the threadId and it was the same for all 3 threads.
So either all threads where doing something weird or I had one thread doing intervals very wrong.

Somehow Netbeans deployment annoyed the server. For when I restarted it without deploying a new version, the services actually kick in right. I also tried to do a classic boolean-switcher from the bean that scheduled the tasks, don't really know if it helped or not. But my intervals was working now.


2: I @Inject my facades to the database (Netbeans Facade-pattern, almost like DAO).
First run worked, but the second run had lost its instance.

So after some masterful reading, googling and swearing I found a soultion, I needed to use the InitailContext and get the @Named bean from there.

yourInstance = InitialContext.doLookup("java:global/YourAPP/YourBean");

And then it was all downhill from there.

I got a lot of experience with Calendar and Date, almost every method in Date is deprecated now, with Calendar as a replacement. So I only use Date as date-instances but for manipulating the date I use Calendar. Go Java 8!


So happy coding! If your having any trouble with a scenario like mine, don't be afraid to ask in the comment-field.

Java EE and Unit testing, Arquillian to the Rescue! Even with Persistence.

As most programmers know, Unit testing is a good practice. You get a quick result if you broke something vital.

And in a simple POJO (Plain Old Java Object) it's not that difficult to make a Unit test, most IDE has JUnit integrated in them.

But what about JavaEE? Now you have a servlet-container, context and persistence to worry about.


There are several solutions for this, the one that I've been focusing on is Arquillian.
Arquillian is a JBoss Community project and development is sponsored by Red Hat, Inc.
http://arquillian.org/

It's a good idé to use Maven with Arquillian. One of the reasons is that you can have different profiles for different servers (glassfish, wildfly, embedded, remote, etc).

My opinions of Maven: http://java-viking.blogspot.com/2014/05/java-ee-maven-and-gradle.html

So Arquillian can simulate JARs or WARs. You can add config-files and classes (or packages).
It can even do persistence.

So i use my Datasource from the server, simulate a War and test it remotely to the server.
The test begins with making a transaction and adding values to the database, I do this with a Facade that Netbeans like to create for JPA Entities.

I've done a test with the findAll-method to see if I get a list with the correct size, and it worked lovely.

So now it's time to test all the important stuff. On my next project I will start with the testing-stuff first now that I know of its capabilities.


So what's next?

Next is to test Arquillian Drone. The Drone-extension is for testing the actual site. So my JSF-sites can actually have automated tests. The newest 2.0.0 (alpha) can even test for mobile.

The Drone can find elements by id and actually do stuff, like login in and testing what the response site is.

So my hours of re-deploying and testing the same thing over and over again feels like a bitter cup of tea right now.

So thats it for now, I have a lot of testing to do. I will also experiment with the managed and embedded server options. But it's always good to have a "remote" server that actually is the same as the release environment,


Comment if you need help with Netbeans, Wildfly and Arquillian, the IDE in the tutorials is Eclipse and uses EclipseLink instead of the Wildfly default (hibernate),

Java EE, Maven and Gradle

So I've been looking a bit at Maven and Gradle.
Gradle is an awesome build tool that have support for the dependencies from maven repositories.

But I have a weak moment, I like the automated stuff in IDEs. And the Gradle-support for Netbeans is not perfect. You can create and open projects, but for a Java EE project, you lose some of the support.

An example is JSF. When you create a normal Web Application in Netbeans, you get automatic support for all the Web technologies it supports. But with Gradle i have to do everything manually.

This is both a good and a bad thing, the good part is to actually learn all the components that are involved, the bad is that it takes time from the coding and frustration while finding the right solution.

Maven, the old and wise, have a different problem. It's problem is that it needs the dependencies declared and downloaded (by maven) while coding.

When you code Java EE against a server, you have the libraries IN the server, the server plugins usually solves this so you don't need to worry about the libraries. They will be provided.
If your using third party libraries, you might need maven? You usually either set the libraries as modules on the server or deploy them separately example Database drivers (JDBC connectors) .


In maven you tell the dependencies to be provided, but you still need them downloaded while coding.
And then comes the kicker, the libraries that the server has, are some times more up to date then those from the maven central repositories!

Even from the same company as the server!

So I'm coding for the Wildfly 8.0 Final, and javax.mail-library isn't in the dependency jboss-javaee-spec. So I had to find it manually and hope that the different version work (1.4.5 in maven, 1.5.1 in wildfly).


So for setting up and starting to code, it's much faster with a IDE and Server-plugin then using Maven.
However!
Maven gives you more structure for different testing and lets face it, many third party libraries are easier to setup via maven. And to automatically do unit-testing in the build face is a plus.

Conclussion:
So I wish that Gradle would be better with Netbeans and I will use Maven for my projects.

JSF 2.2 FileUpload and a way to get it back

So my Web app needed to be able to upload images for users, "connect" the user and the image and to be able to actually show the image.

Upload:

So JSF 2.2 has a really simple way of doing this. You just put <h:inputFile /> and connect this to a Bean. The type should be "Part" (javax.servlet.http.Part). But to remember is that the <h:form> needs too add enctype="multipart/form-data" as an attribute, <h:form enctype="multipart/form-data">.

Connect:

So now we have a Part in the Bean, but how to save it and where?
Some may want to save the image in the database, this can be done with the BLOB-type. you then have to get the input stream and read it as bytes.

Pros:
All in one place, handled by the DBMS.

Cons:
Database takes much more space
Harder to export all the data for backup-solutions

So what do we do then?
We can save it on the hard drive as regular files. A risk is if the file contains a virus.
How we actually save this is rather easy. The Part-class has a "write"-method which you either can give a relative path or a absolute path. The relative path may put it in the web-applications tmp-folder, it's really the server who controls where its put.

The absolute path makes it easy to do backup scripts to make backups. The big problem is how to reach the files from the web app.

Path:

How to reach files from "above" the web-apps folder is different for the servers. Glassfish has Virtual host/server. Some has context-properties.

The one I have focused on is Wildfly (JBoss Application Server 8).
It was a bit tricky to find the right answer.

But in the config file for your server (domain.conf or one of the standalone confs) there is a subsystem for undertow. Undertow is the new servlet container for JBoss.

In that subsystem there is two parts that's interesting for us right now: Host and Handlers.


In the Handlers part we add
<file name="images-handler" path="your/path/to/save"/>

you can use your servers path-variables

and in the Host part
<location name="/images" handler="images-handler"/>

The handler takes care of where the folder will be retrieved from (read from) and the Host will controll how the web-app will reference it.
In this case you reach it by http://serveradress:port/images

Display:

Now comes the last tricky part. The images-folder is reachable now BUT is reachable by the server-adress WITHOUT the web-apps adress on top of it. You can probably map it to be reachable.

So <graphicImage> gets problems due to it being relative to the web-app.
A simple <img> works great thou.


So either find a way to map it better or use a regular <img>


So that's it for the simple fileUpload of images, especially with Wildfly.

Code ahoy!

JPA: Hibernate Implementation "lazily initialize a collection"

Today I have struggled half a day with a frustrating problem.

I have a ManyToMany-relationship (JPA) with two Entities. Shop and Brand, it's also bidirectional.

Hibernate and Spring-veterans sees where I'm going with this...

If I want to get a Object of Shop from the database, everythings simple. But if I want to get the attribute that's the "ManyToMany" (in this case "Brand") I would get it as a form of Collection.

But here's the kicker: Hibernate has a lazy initialization of these kinds of collections. This is for saving a lot of data loading for every time you get a Shop-object from the database.

I wanted to have a JSF-form with my data and the possibility to update it (specifically the Brands). I wanted to use the selectManyCheckBox component in JSF.

So the Hell began here.

The specific exception I got was (I also got the role-error earlier):
javax.servlet.ServletException: failed to lazily initialize a collection, could not initialize proxy - no Session

You can set the annotation of the ManyToMany to suggest "EARLY" so that it loads all the collection-data with the regular.

And that solution didn't solve my problems. It feels like I almost tried every solution known to man to solve this problem. I did beans that sole purpose was to get the Brands as a list from the Shop and try to merge it with the data in the form and the update-bean.

Of course I tried to google solutions. 95% of the links led me to thinking it was the classic lazy-problem that the Hibernate-session closed before I got my collection data.

But something wasn't right. Because I actually got the data to show in the form correctly quite early. It was the update that caused the problem.

The update was a bean that held the data from the JSF-component, they were connected.
But why did a lazy-error show when it wasn't supposed to load the data but actually update it?

I tried View filter (Open View in Session-pattern), I tried Stateful beans with EXTEND persistance unit.


After half a day I stumble to Hatim Alimam's blog were he had the same problem (but with Primeface implementation).
http://blog.hatemalimam.com/hibernate-lazyinitializationexception-and-pselectcheckboxmenu/

It was a little <f:attribute> that was needed!
If I just put   <f:attribute name="collectionType" value="java.util.ArrayList" /> in the <f:selectManyCheckBox>    Everything worked!

So it wasn't really the classic Lazy-problem, but with that Exception-message, one might think so...

So I hope anyone that has the same problem I had either stumble to this blog or Hatim's.

Cheers!

Wildfly Injection and JPA: troublesome

So I've been making a Webapplication with classic CRUD via JPA. The database is also used for login-functions. I use JSF with Facelets for View.

JSF can manage beans easily e.g (value="#{myBean.value}")

So I thought that the Bean could be a @Entity. I even have a book that use it like that.
I then Inject the bean in my Facade (a DAO) that should persist it.

I checked so that it had the values that I gave from the View (JSF-form).


But here's the kicker!

I get "java.lang.IllegalArgumentException: Unknown entity: myclass#$Proxy$_$$_WeldClientProxy".

I tried of writing the classes manually in the persistance.xml without any luck.

After a lot of digging, I learned that the Bean that Weld injects (I use Wildfly-server that uses Weld) you doesn't actually get a real instance of your bean. You get a Proxy. So I could get the values, but it was not seen as an Entity.

I confirmed this by making a manual instance of the bean and persisting it with the same Facade-code.

The only difference was the Injection.

So my solution was to make a controller bean which had a manually created field of the bean. And use that bean e.g (value="controllerBean.myBean.value")

The controller bean then used the Facade for persistance.

This solution works for me, the controller bean can then do some validation and more.

Fine and dandy!

Wildfly, Impressions and configs

I'm using the Wildfly Java server for my current projects. I have a little Raspberry cluster of them even (I'm going to update the post how to set them up).

So how do I feel about Wildfly?
It's more configuration then programming! But after some talking to developer-friends, that's actually much of Java EE in the beginning.


So the latest part of the project was to make a Login-function. I've made a simple function like this in PHP. So I thought "what the hell".

After some research I found that Java EE hade it built in the servlet (you can even hide it in a simple html-form if you name the components just right).

So I tried to make a "customized" login which would use a form rendered by JSF and sent the values to a JavaBean which would use the servlet-login method.

Sounds easy? Yeah, that was my weekend...

By using built in functions, you got to know all the magic that actually happens, which isn't the same for every Java server, but at least similar.

First you have to choose if your going to save the users in a file, database or LDAP.
You then need to setup the security-domain to use the right options. For me, I needed to use databases,
That meant to connect it to the right JNDI (Datasource for the Database) and this was actually the last piece of the puzzle to fall into place. Many sites write a JNDI that seems to differ from the regular, but nothing is different! Just use the regular JNDI.

For database-solution you need to save the users (passwords and all) and the different roles they have in the database. You then set the queries for users and there passwords in the same security-domain.

For Wildfly you then need to create/edit the jboss-web.xml file to set it to use this security-domain (the one you configured).

You can also set the encryption and hash-algorithm for the password in the security-domain.

Then comes the web.xml configuring. At least this is the same for the most servers.
Here you get to set which options and which pages will and role will demand login and which site is the login-site and an error-site for getting the error-message (can have different for different errors).

I haven't puzzeled together all the pieces and don't really know all the requirements yet, but hopefully I will puzzle it out soon.

Hopefully this is one of the bigger obstacles I will face for my little project.


Impressions of Wildfly so far:
Fast and has many options (standalone, standalone-full, domain, etc).
Neat admin-console, and a good CLI-console.
Bad error-messages from admin-page (unknown error) is a regular occurence.
New server so it doesn't have many sources with examples yet,
Lots of configurations that needs to be done, some which you can't do from the Admin-page.

Java-Cluster: Wildfly and User Configuration

Now for the Tricky parts:

First lets add a user to the raspberry for the Wildfly (we don't need to make it a sudo (administrator) user).

"sudo useradd -c "JBoss AS User" -s /bin/bash -d /home/jboss -mr jboss"

The users name is "jboss".We should then give ownership of the wildfly folder to this user
"sudo chown -R jboss:jboss /usr/jboss/wildfly-8.0.0.Final"

It's also a good idea to give the user a password:
"sudo passwd jboss"

Now lets switch to that user:
"sudo su - jboss"

First of all we need to change the memory usage of the Wildfly-server. Per default it uses a maximum of 512mb (which is ALL of the memory the Raspberry Pi has! And we need some memory for other stuff).

So edit the file /usr/jboss/wildfly-8.0.0.Final/bin/domain.conf, I usually use nano for editing (you can use vi or something else if you prefer it)
"nano /usr/jboss/wildfly-8.0.0.Final/bin/domain.conf"

Somewhere in the file there is a line "-Xms64m -Xmx512m -XX:MaxPermSize=256m"
change this to:
"-Xms64m -Xmx196m -XX:MaxPermSize=256m"

Exit nano using CTRL + X and choosing "y" to save the changes.

Now we can try to start the server, go in to the Wildfly bin-folder and run:
"./domain.sh"

It may show some red text but wait and see if it starts.

Do these steps for all the Raspberry Pi's that should be part of the Java-Cluster.

Now it's time to choose a Master. One of the Raspberries have to be the Master, the rest will be "slaves".

Ssh into the master

In the Master we will now create som Wildfly users. In the Wildfly bin-folder of the master:
./adduser.sh

Pick a Username (like "master") and a password
Choose the Management option

The Master-user don't need a secret value (the last question the scripts asks)

Now add a user for every slave (In the Master). Give them individual names, choose Management option and answer yes to the last question.
Copy the value to a file with the user names. Example:

master
slave01 <secret value="THE VALUE IT GETS" />
slave02 <secret value="THE VALUE IT GETS" />

Now lets config The Master: edit the file host.xml in the domain/configuration folder:
change:

<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
    </interface>
    <interface name="public">
       <inet-address value="${jboss.bind.address:127.0.0.1}"/>
    </interface>
    <interface name="unsecured">      
       <inet-address value="127.0.0.1" />  
    </interface>
</interfaces>

To: where Masters IP should be the IP of the master

<interfaces>
    <interface name="management"
        <inet-address value="${jboss.bind.address.management:Masters IP}"/>
    </interface>
    <interface name="public">
       <inet-address value="${jboss.bind.address:Masters IP}"/>
    </interface>  
    <interface name="unsecured">
       <inet-address value="Masters IP" />  
    </interface>
</interfaces>

That's it for the Master!

Now the steps that needs to be made for every slave:
Ssh into the slave

Edit the host.xml in domain/configurations of the wildfly-folder

change:
<host name="master" xmlns="urn:jboss:domain:1.1">
to:
<host name="slave01" xmlns="urn:jboss:domain:1.1">
where slave01 is the name of the management-user created in the host for this slave

Then we need to modify domain-controller section so slave can connect to master's management port (in the same file):

Change it to:
<domain-controller>
   <remote host="Masters IP!!!" port="9999" security-realm="ManagementRealm"/>
</domain-controller>

Change the interface part to:
<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:SLAVE IP!!!!}"/>
    </interface>
    <interface name="public">
       <inet-address value="${jboss.bind.address:SLAVE IP!!!!}"/>
    </interface>
    <interface name="unsecured">      
       <inet-address value="SLAVE IP!!!!" />  
    </interface>
</interfaces>

Chane security-realm part: Chane the value of secret to that you got from creating users in Master.
comment out <local default...

<security-realms>
   <security-realm name="ManagementRealm">
       <server-identities>
           <secret value="THE SECRET VALUE of SLAVE"/>
       </server-identities>
       <authentication>
           <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
       </authentication>
   </security-realm>
</security-realms>

Now repeat for every slave.

Now test to start the master with ./domain.sh
When its started, try to start the slaves the same way.

Check the information on master, if it registers the slaves then it works!

In the next part I will discuss my results of trying a simple web-project and deploying it.

Code ahoy!

Java Cluster: Software SD-card Part

In this part I will explain all the steps for installing the system, the next part will be configuring Wildfly.

First you need a SD-card slot/reader.

Second Download Rasbian from: http://www.raspberrypi.org/downloads

See http://www.raspberrypi.org/wp-content/uploads/2012/04/quick-start-guide-v2_1.pdf for instructions on how to prepare your SD-card and get the image to it.

Repeat with all 4 (you can probably make a script for automating all these tasks but I'll keep it simple)

My tip is to power up one raspberry at a time. Because my router gives the same IP-number to the devices, and if we power up all 4 I would have 4 devices named raspberry.

So I start one, change its host name and save the IP-number in a file with the hostname for later referencing.

So start the first and find out it's IP-number.

Ssh into the raspberry with: "shh pi@<ip-number>" the password is rasberry

run sudo raspi-config:

1. First choose "Advanced" and "Update" so the setup-program updates to the latest version.
2. Expand the Filesystem so it uses the whole SD-card
3. Change password (your choice)
4. Choose "Advanced Options" and memory split. Give the GPU 16 (instead of 64) we don't need so much memory for the graphics.
5. In "Advanced" you can also change the hostname. I choose to have a system like cluster01 through cluster03.
6. I choose to overclock mine to 900mhz but It's up to you.
7. Choose "Finish"

My Raspberries usually complain on the locale, but we can fix it by editing the Locale manually:

"sudo nano /etc/default/Locale" and adding these lines to the file:

LC_ALL=en_gb.UTF-8

LANGUAGE=en_gb.UTF-8

After that run "update-locale", don't worry if it still says some bad stuff.
Reboot by "sudo reboot" then ssh to it again.

Do a full system update
"sudo apt-get update" then "sudo apt-get upgrade"

Then install oracles java 8
"sudo apt-get install oracle-java8-jdk"

You can log out of the ssh for a little while now

Now its time to download Wildfly 8
http://wildfly.org/downloads/

I choose to download it on my desktop computer and copy it via scp to the raspberry
(used via Linux on my desktop in the folder where Wildfly zip is)
"scp pi@<ip of raspberry>:. Wildfly-8.0.0.Final.zip"

this copies it to the home-folder of the user "pi".

Ssh into the raspberry again

Make a folder in the /usr with the name jboss where we will unzip the zip
"sudo mkdir /usr/jboss"

Then by standing in the folder which the zip is (check with "ls")
"sudo unzip -d /usr/jboss /home/pi/wildfly-8.0.0.Final.zip"


Repeat all the steps with all 4 (you can probably make a script for automating all these tasks but I'll keep it simple)

It's now time for configs!
(Continued in the next Part)

Java Cluster: Hardware

I'll first show the Hardware setup then later show the Software configs.

The hardware is:

4 x Raspberry Pi Model B rev 2 (512mb)

4 x Raspberry Pi Heatsink (keeping it cool!)

2 x Stackable case (https://www.modmypi.com/blog/multi-pi-assembly-guide)

4 x Short Networkscables

4 x short Usb-Micro Usb

1 x Deltaco 7 port Powered Usb-hub (http://www.dustinhome.se/product/5010144399/deltaco-usb-hub-7-portar-usb-2-0-svart/)

1 x 5 port switch

4 x SD-cards (8gb or higher class 10)

First set the Heatsinks (cooling paste is already applied)

I choose to angle them sideways if I want to add a fan at the side.

Next is to screw the Raspberries to the cases and stacking them. 

Connect the Network cables to the switch, the usb cables to the usb hub. 

Connect a cable from your modem/router to port 1 of your switch (should give all your raspberries access to Internet).

All that's left is fixing the SD-cards. But that is a part of the Software explanation.

(Turn on the power AFTER all the SD-cards are fixed)

Cluster Time!

Cluster Time!

I'm not the most knowledgeable person when it comes to hardware and networks. Hey I'm a programmer, I work with software :)

But for some reasons I wanted to make a Java-server-solution at home. I'm not a big fan of having a computer with a loud fan going at it in my living room. So what's the options?

I actually have a mini-ITX motherboard with passive cooling, but the parts have started dying, first the nic and now the display connection.

So what's the current solution for every nerd? Raspberry Pi!

So I went to the local hardware store and bought yet another one of these little cuties. (I have one for streaming media from my NAS, awesome that I can control it via the TV-remote through HDMI).

Rasbian (Debian Wheezy for Raspberry Pi) was the distro of my choice. It has Oracle Java (even the new 8!), It didn't take long to get a Glassfish-server up and running. And with a little wrestling with the router I made so I could reach a website (written in JSP) from the web.

The problem was the CPU. Even with the overclocked option to 900 mhz the cpu got upp to 85-95% with little demand.

So what to do now?

I'm still just in the beginning of my JavaEE learning quest. But one thing that has caught my attention is Wildfly (JBoss newest server). I wanted to see if it was more lightweight then Glassfish (the opinion seems to be that ANYTHING is faster then Glassfish).

But is this nerdy enough? Short answer: Of course not!

So what can I do to make this better? Cluster time!

Cause Raspberries are so cheap I could by several and get them in a cluster, challenge accepted!

I will explain how to actually do this in a future post. I'm missing another stackable chassi (out of stock at the store).

Al the raspberries have heatsinks on them. They (3 soon to be 4) are powered by a usb hub (powered) and is connected to a switch which is connect to the hub.

One of these will be a database-server the other ones will be master and slaves with Wildfly.

Right now to Master-slave is done but have not been tested with any applications.

Coding ahoy!

To be continued...

Intro

Hi

This blog is going to be mostly about my growing knowledge of Java and some nerdy projects I'm doing. English isn't my native language so I will use this opportunity to practice it.

If it's a demand I can do the blog in both Swedish and English

/Karl