18 days ago

The Vaadin-on-Kotlin framework contained the support for feeding the outcome of a SQL query to the Vaadin Grid for the long time. However, since VoK is based on Kotlin, Java developers probably avoided this solution and looked for a Java-based example. Well, here it is:

Both of the examples are pure Java, even though the project uses code from Vaadin-on-Kotlin. The Kotlin stdlib is only included as a run-time dependency - the example projects contains no Kotlin code and doesn't even run the Kotlin compiler.

The example projects demo the following features:

  • A full-fledged implementation of the SQLDataProvider, which supports paging, sorting and filtering.
  • A FilterBar for Grid, with auto-generated filtering UI components, passing filters to the SQLDataProvider. This is a direct replacement for the FilteringTable Add-On, but it targets Vaadin 8 Grid as opposed to FilteringTable targeting Vaadin 7-compat Grid.
  • The FilterBar is also available for Vaadin 10, targeting native Vaadin 10 Grid.

The example projects use Maven as the build system.

The SQLDataProvider connects to the database directly, using the VoK-ORM library; by default a direct access to a JDBC DataSource is required, and the JDBC connections are pooled using the Hikari-CP connection pooler. However, if you're running in a Spring or JavaEE container, you can override this behavior, see Vok-ORM documentation for details.

Edit: JPADataProvider

For those of you who just can't let go, I've prepared a full-fledged JPADataProvider example, with filtering, sorting and filter bar: Vaadin 8 JPADataProvider Example with a live demo

 
3 months ago

While I don't recommend using Spring because of simplicity reasons, there are valid reasons to have Spring support. For example when building UI for an already existing Spring-based backend, or when gradually converting a legacy application to VoK.

Use global service registry

The simplest way to go is to build a global service registry: a plain class with static getters which retrieve particular services from Spring (e.g. using ApplicationContext.getBean() call). To obtain the ApplicationContext just inject it into your UI and then simply call UI.getCurrent().getAppContext().getBean(). This approach has the following advantage over field injection approach:

  1. You can call service getters from anywhere you want; you don't have to trouble yourself with constant worry whether the view/component in question was constructed by Spring and has everything injected properly. This fixes a lot of UninitializedPropertyAccessExceptions you might have.
  2. You can even build the registry as a set of extension methods to a Services object as recommended by Vaadin-on-Kotlin. Please read Writing Services for more info.
  3. You don't have to worry whether the injected Service Proxy is serializable and whether it will be reinjected upon deserialization (it won't, a constant cause of NullPointerExceptions).

Injections

The harder way is to use injections. If you decide to go this way, your views must be constructed by Spring so that they have all dependencies injected properly. To achieve this, you must not use the autoViewProvider + @AutoView VoK mechanism (since AutoViewProvider constructs views simply by calling their constructors). Instead you must use SpringViewProvider and annotate your views with @SpringView.

Often you build custom components which depend on Spring services and use injection to obtain service instances. Such components must also be instantiated with Spring otherwise no injection is performed. To achieve this with the Kotlin DSL just write something like this:

fun (@VaadinDsl HasComponents).appToolbar(block: (@VaadinDsl AppToolbar).() -> Unit = {}) = init(UI.getCurrent().appContext.getBean(AppToolbar::class.java), block)
 
5 months ago

I'd like to write something positive, but I can't - my experiences with CouchBase had been highly negative. I'd even dare to say it's an unfinished piece of junk; if you need to setup a syncing platform from your Android app, avoid CouchBase like a plague.

At first everything was peachy, the synchronizers worked correctly and things were synced to the server. Then everything went south.

I've noticed that CouchBase Server permanently uses 100% of one CPU core. It has been explained that it's because it gathers statistics in the background. Apparently gathering statistics of an empty bucket (0 documents) is highly time-consuming in CouchBase world. That should be a warning sign of the quality (or rather, the lack thereof) of the whole CouchBase offering. That, combined with a constant stream of poor excuses from the devs for every bug I've opened. I should've abandon the CouchBase stack, but it seemed to be working so far, so I thought I'm simply doing something wrong.

Then things started to break.

One does not simply use CouchBase server from Docker Compose

At some point the CouchBase Server database grew large enough (well not really that large, 2 million documents is not a big database) so that it started to take 2 minutes to start the Server. However, CouchBase Sync Gateway apparently is short-tempered and gives up after a minute or so, simply terminating. That makes it impossible to launch a CouchBase Gateway from docker compose, simply by calling docker-compose up since Gateway will die.

I had to write a script which starts the CouchBase server, then sleeps for a minute and then starts the Gateway. Dumb.

NoSQL running on a SQL?

I was worried about the performance on Android, since CouchBase Lite uses sqlite as a backing storage on Android. Brilliant - NoSQL database running on top of SQL, which combines the slowness of the SQL with zero ACID guarantees of the NoSQL! However, it was promised that the Forest DB would perform 2x to 5x faster and thus I gave it a try. It was not a good idea.

Apparently ForestDB is not finished or is flaky or broken, since it gave weird results on the old map/reduce queries for list keys. You know - you search for all keys of type List with first item equal to 1 and you'll get lists starting with 2, 0 and others. I couldn't even reproduce those issues since they happened randomly. Also, I could mention the docs which were totally unclear as what will actually be selected, but I can't - CouchBase Lite documentation got rewritten and apparently now every query should use SELECT-like queries, brilliant! Talk about backward compatibility.

To test for flaky query API I've decided to write tests for those views. A bad idea.

When One close() Fails

I've written simple JUnit tests which opened a database, did a bunch of writes/reads and then closed the database. I noticed that those tests ran pretty slowly; the culprit was the database closer which took like 30 seconds to close the underlying ForestDB database. Upon closer examination I was surprised to find that the native call to DB close actually failed. A brilliant solution was devised - if the DB close fails, repeat. And repeat. And repeat. If it keeps failing, try at most a hundred times, then consider the database closed. Who in the name of God would consider this a good idea?

Anyway, from the point of view of CouchBase Lite devs everything is fine now because they've dropped ForestDB and no longer support it in their offerings. So I'm stuck with half-assed junk which is no longer supported and will never be fixed, and can't even migrate to the sqlite backend. Brilliant.

https, https everywhere!

Securing CouchBase Gateway is simple - just turn on the https in the Gateway config and provide a DNS name. What the docs won't tell you is that by doing that, CouchBase Gateway will reject all requests which do not target that particular DNS name. You no longer can access the Gateway using localhost, an IP address, a docker compose alias from within the docker container - you must use the DNS name. And sometimes you can't.

In a moment of blissful ignorance I've decided to upgrade Ubuntu 16.04 to 16.10 on Leaseweb. A bad idea - the networking stack got borked in some incomprehensible way that the Gateway client app running on that very machine in docker couldn't access the Gateway through the DNS name. The connection would simply be rejected for no apparent reason. I failed to find the reason, and after I complained, Leaseweb offered to look into the issue (no promise of solving the issue, they would just look) for a mere $100 per hour.

So there I was - the software stack crumbled and my app was unable to connect to the Gateway.

The Sync No Longer Syncs

I'm using Document attachments quite heavily. Basically every document had a short 2kb binary data attached and I did not wanted to use base64 to attach the binary to the document's JSON, so I've used attachments. Worst idea ever.

I've noticed several of the Document lacking attachments - suddenly after sync the attachments became missing, CouchBase Lite would simply return null and my code would then fail since it expected a non-null binary (which should have been there in the first place!). Since Gateway has no state, the culprit is probably the CouchBase server itself. This conclusion is reinforced by the Sync simply failing because of getting 500 INTERNAL SERVER ERROR from the Gateway, which is apparently also confused where the hell those attachments vanished.

However, the outcome of this issue is that the Sync no longer works - every sync attempt is simply killed by 500 SERVER ERROR so nothing syncs, and the database is corrupted by missing attachments.

Conclusion

The CouchBase software stack betrayed me so many times I've stopped counting. It wastes CPU and electricity, it silently corrupts the data, the code is horrible and stupid code practices are common. I consider the whole offering flaky and won't use it anymore.

CouchBase sucks, don't use it.

 
6 months ago

Creating Vaadin UIs from Java code has the following disadvantages:

  • The UI structure (how the components are nested into each other) is not clearly visible from the code
  • There is no mechanism to enforce the component configuration code to be grouped together in one place. If the programmer is not careful, the config code for different components may mix

Consider the following code:

class WelcomeView: VerticalLayout(), View {
    init {
        val button = Button("Save")
        val formLayout = FormLayout()
        val name = TextField()
        val age = TextField()
        name.caption = "Name:"
        age.caption = "Age:"
        formLayout.addComponents(name, age)
        addComponents(button, formLayout)
        button.icon = VaadinIcons.CHECK
        button.addClickListener {
            Notification.show("Saved!")
        }
    }
}

The code is a mess of random assignments, with no clear structure. The first improvement we can do is to use the .apply{} function to group the component initialization code:

class WelcomeView: VerticalLayout(), View {
    init {
        val formLayout = FormLayout()
        val name = TextField().apply {
            caption = "Name:"
        }
        val age = TextField().apply {
            caption = "Age:"
        }
        val button = Button("Save").apply {
            icon = VaadinIcons.CHECK
            addClickListener {
                Notification.show("Saved!")
            }
        }
        formLayout.addComponents(name, age)
        addComponents(button, formLayout)
    }
}

Much better. Now the component initialization code grouping is actually enforced by the compiler. However, there is much to improve; for example the UI structure is still not yet visible. What if we move the TextField initialization code into the parent layout .apply{} block?

class WelcomeView: VerticalLayout(), View {
    init {
        val formLayout = FormLayout().apply {
            val name = TextField().apply {
                caption = "Name:"
            }
            val age = TextField().apply {
                caption = "Age:"
            }
            addComponents(name, age)
        }
        val button = Button("Save").apply {
            icon = VaadinIcons.CHECK
            addClickListener {
                Notification.show("Saved!")
            }
        }
        addComponents(button, formLayout)
    }
}

Now the structure emerges, but the components are inserted into parents in a reverse order (button before the form layout) which is not what we want. We want the components to be inserted in the very same order in which they are create in the code. When we create a component, we typically want to add it to the "parent" layout immediately, and we can take advantage of this:

class WelcomeView: VerticalLayout(), View {
    init {
        addComponent(FormLayout().apply {
            addComponent(TextField().apply {
                caption = "Name:"
            })
            addComponent(TextField().apply {
                caption = "Age:"
            })
        })
        addComponent(Button("Save").apply {
            icon = VaadinIcons.CHECK
            addClickListener {
                Notification.show("Saved!")
            }
        })
    }
}

Getting there, but the code is quite chatty. Maybe we could rewrite the addComponent(), Button() and .apply{} into one function? The function must know into which layout the button is going to be inserted; also the function must run the configuration block so that we don't have to write the .apply{} ourselves. The first prototype could look like this:

fun button(parent: ComponentContainer, caption: String, block: Button.()->Unit) {
    val b = Button(caption)
    parent.addComponent(b)
    b.block()
}

If we write similar functions for FormLayout and TextField, that will allow us to write the code as follows:

class WelcomeView: VerticalLayout(), View {
    init {
        formLayout(this) {
            textField(this) {
                caption = "Name:"
            }
            textField(this) {
                caption = "Age:"
            }
        }
        button(this, "Save") {
            icon = VaadinIcons.CHECK
            addClickListener {
                Notification.show("Saved!")
            }
        }
    }
}

Better, but we keep repeating the this parameter. this always points to the current parent layout where we want to add the components. And there is a way to transfer this automatically into the function itself - by defining the functions as an extension functions on the layout:

fun ComponentContainer.button(caption: String, block: Button.()->Unit) {
    val b = Button(caption)
    addComponent(b)
    b.block()
}

Now the code is perfect:

class WelcomeView: VerticalLayout(), View {
    init {
        formLayout {
            textField {
                caption = "Name:"
            }
            textField {
                caption = "Age:"
            }
        }
        button("Save") {
            icon = VaadinIcons.CHECK
            addClickListener {
                Notification.show("Saved!")
            }
        }
    }
}

To use this approach with Vaadin, simply use the Karibu-DSL library - it introduces such extension functions for all Vaadin components for you, allowing you to build your UIs in a structured way.

 
6 months ago

Proper building construction is to have a proper design which includes all latest state-of-the-art components and follow all safety precautions. The building won't collapse and it will be great. If you however use this approach with software, you will fail.

The thing is, when the building is done, it is done. Nobody will suggest to add more elevators or change the grounding. A total opposite to a software which is an ever-changing reshaping blob. A living, breathing organism where anything may be subject to change. It is an act of evolution, as opposed to the construction's creationism.

By using lots of frameworks in the software project, albeit with best intents, you will create bonds which artificially slow down the shape-shifting. Too many bonds, and the software will stop moving, it will die, it will start rotting and it will turn into a rotting steaming spaghetti of framework workarounds and annotation magic. Your devs will waste time figuring out how to get around the framework limitations. Good developers will get frustrated by the slow pace and they will leave. The software will attract framework fanatics. Under the guidance of bad developers piling workarounds upon hacks, the team will repulse any good programmers. The software ecosystem now reached terminal state and the software now starts to rot. Good luck finding a team of good programmers that can now restart your project afresh.

To avoid software rot, be careful when adding frameworks, and remember to remove them when they no longer serve any purpose and deliver no value in your software. Only by focusing on simplicity, you will be able to evolve the software quickly.

Remember: Simplicity is the best future-proof that you can have in your software.

This is the most important guiding principle in the Vaadin on Kotlin framework.

 
7 months ago

The client-side in Vaadin 10 is completely different than the one in Vaadin 8 and the layouting manager from Vaadin 8 is completely gone. Vaadin 10 layouting is therefore something completely different to Vaadin 8 layouting. Naturally, it is something completely different to Android layouting. Here we will focus on layouting completely configured from server-side, and not from a CSS file.

For Android Developers

  1. There is no RelativeLayout.
  2. There is no AbsoluteLayout yet.
  3. There is VerticalLayout and HorizontalLayout to replace LinearLayout
  4. To replace ListView, create Vaadin Grid with one column, containing components as contents. The content will be lazy-loaded and the components will be constructed lazily. The issue here is that Vaadin Grid requires all rows to have exactly the same height (or at least it used to).
  5. There is no GridView. You can employ FlexLayout which allows for multi-line horizontal layouting, but there is no lazy loading and the components will be instantiated eagerly.
  6. There is no table layout supported out-of-the-box by Vaadin 10. You can try to employ css grid but it's quite new and only supported by latest browsers.

For Vaadin 8 Developers

  1. There is no AbsoluteLayout yet.
  2. There is no GridLayout. You can try to employ css grid but it's quite new and only supported by latest browsers.
  3. There is a replacement for VerticalLayout and HorizontalLayout but it behaves differently. Read on for more info.
  4. There is FormLayout and it is responsive. See the vaadin-form-layout element documentation for more details; use FormLayout class server-side.

Let's learn VerticalLayout and HorizontalLayout

The very important rule #1:

  1. There are no slots allocated for child components. Because of that, calling setSizeFull() or setWidth("100%")/setHeight("100%") on children will not fill in the slot - instead it will set the component to be of the same width or height as the parent layout is. Hence it will most probably overflow out of the parent layout. Therefore, never call setSizeFull() nor set width/height to 100% unless you absolutely know what you're doing. When you set the component to expand, it will be automatically enlarged.

The new HorizontalLayout and VerticalLayout are classes designed to mimic the old Vaadin 8 layouts as much as possible. Yet underneath they use a completely different algorithm so you can't expect a 100% compatibility. The new algorithm is called flexbox and you can learn about its capabilities in the Complete Guide to Flexbox. Don't pay too much attention to the terminology - both HorizontalLayout and VerticalLayout try to use the "older" Vaadin 8 terminology as much as possible, making it easier to understand the functionality.

We will also not use vanilla Vaadin 10; instead we will use the Karibu-DSL library which enhances both HorizontalLayout and VerticalLayout in a way that they are more compatible with Vaadin 8. The simplest way to experiment with the layouts is to clone the Karibu-DSL HelloWorld Application for Vaadin 10 project. Just follow the steps there to get the project up-and-running in no time.

HorizontalLayout

We'll modify the MainView class as follows:

@BodySize(width = "100vw", height = "100vh")
@Route("")
@Viewport("width=device-width, minimum-scale=1.0, initial-scale=1.0, user-scalable=yes")
@Theme(Lumo::class)
class MainView : VerticalLayout() {
    init {
        content { align(stretch, top) }

        horizontalLayout {
            content { align(left, baseline) }

            textField("Enter your name:")
            checkBox("Subscribe to spam")
            comboBox<String> {
                setItems("Once per day", "Twice per day")
            }
            button("Create Account")
        }
    }
}

The focus here is the horizontalLayout's content { align(left, baseline) }. It tells the layout to align the children to the left, and vertically on the baseline:

The baseline setting keeps the components in line and aligned properly on a single line. If we were to change that setting to top, all components would move upwards (except the TextField which is the largest one and is held in-place by its label):

Note how the checkbox is no longer aligned properly with the rest of the components - instead it is glued to the top.

Let's demonstrate the 100% mistake. Let's set the textField's width to 100% as follows:

        textField("Enter your name:") {
            width = "100%"
        }

The result is very weird: it will push other components to the right and shrinks them. I was expecting the first element to be as wide as the parent, pushing all other components to overflow; I was wrong and something else happened :)

Let's now remove the 100% width and let's continue. There are two more vertical alignments: center:

And bottom:

This almost resembles the baseline setting, but the combobox looks misaligned. Therefore, if you're aligning fields, most probably you'll want to use baseline.

There is one more setting which is very important if you compose other layouts in HorizontalLayout - the stretch setting.

Now for the horizontal alignments: there are the following options:

  • left, which corresponds to flexbox's start
  • center
  • right which corresponds to flexbox's right
  • around, between and evenly distributes the leftover space around the individual components.

To see the effect of those settings please see the justify-content flexbox setting.

Of course these settings work only if there is a leftover space. If the HorizontalLayout is set to wrap its contents or if at least one component expands, the leftover space is gone and these settings are ignored.

Expanding

Now let's expand the TextField as follows:

        textField("Enter your name:") {
            isExpand = true
        }

The isExpand property only works inside of a VerticalLayout or HorizontalLayout and it is an alias for setting of the flexGrow to 1.0. This makes the TextField expanded in a way that it will eat all available leftover space:

Of course you can set other components to different flexGrow settings; a component with the setting of flexGrow = 2.0 should roughly take two times the space of a component expanded by flexGrow = 1.0 (this depends also on what the inherent size of the component is).

When there is no leftover space (for example because the HorizontalLayout wraps its content) then the flexGrow/isExpand setting is ignored. However, in this case the parent VerticalLayout is set to stretch its children to the available width, and hence the HorizontalLayout will be set to fill parent and that creates leftover space in order for the flexGrow setting will work.

Note how the components were automatically enlarged, to accommodate the space they've grown into, without any need to set their width to 100%. Setting the width to 100% would break the flexbox algorithm completely and would introduce weird artifacts.

Since Vaadin 10 does not use any kind of layouting engine but the CSS engine, it is actually the browser who is responsible for laying out the children. This means that you can use the browser's built-in developer tools to play with the components - you can set the flexbox properties in any way you see fit. The only disadvantage is that the flexbox terminology will differ from that of HorizontalLayout.

The browser is a very powerful IDE which can help you debug CSS- and layout-related issue. Take your time and read slowly through the following tutorials, to get acquinted with the browser
developer tools:

Shrinking

When the components are so tightly packed that there is no room even for their preferred size, the flexShrink rules are taken into effect. That is however beyond the scope of this tutorial, you can read more about this at The Guide To Flexbox or Mozilla's flex-shrink documentation.

Overriding the default alignment for children

You can override the horizontal alignment per child, by using the verticalAlignSelf property:

        comboBox<String> {
            setItems("Once per day", "Twice per day")
            verticalAlignSelf = FlexComponent.Alignment.START
        }

You however can't change the vertical alignment per-child.

VerticalLayout

The VerticalLayout behaves exactly the same as HorizontalLayout does, it just swaps the axes and lays out the component along the y axis.

Root Layout

You can use both VerticalLayout and HorizontalLayout as root layouts. Typically you set them to span the entire browser window by calling setSizeFull() - that's the only place where you should use the setSizeFull() call :-).

Of course these layouts are quite rudimentary and may not have enough expressive power to define the general UI of your app. As you grow accustomed to these layouts, you can then switch to FlexLayout which offers you all possibilities of the flexbox. See the mobile-first header/footer example in the Complete Guide to Flexbox article.

 
7 months ago

There are lots of Android SDK sucks and android development is a pain rants indicating that development on Android is a horrible experience. Instead of repeating the same thing over and over again, I'll try to sum up why I will definitely prefer Vaadin-based PWA for my next project over Android SDK:

  • Vaadin has no Fragments - no crazy lifecycle of create/start/resume/whatever. The app simply always runs. In Vaadin, components are used for everything; nesting components is perfectly fine as opposed to nesting fragments, which tend to produce weird random crashes (the famous moveToState() method).
  • Since in Vaadin there is no crazy lifecycle, you are not required to code defensively. You do not have to be always prepared to be passivated by the Android system; you do not need to shatter your algorithm into multiple methods, always having to be prepared to save state into some Bundle when Android decides to passivate your app. Google guys are apparently obsessed by passivation and making your code fragmented, defensive and really hard to maintain.
  • Vaadin components are lightweight - they don't use any native resources, they just produce html for the browser to draw. There is no complex lifecycle - the components simply attach and detach to the screen (represented by the UI class) as the user uses the app. Since the components do not use any native resources, they are simply garbage-collected when not needed anymore.
  • All UI components are Serializable; in a rare case when the session passivation is needed, they are automatically saved along with the http session when there are no requests ongoing.
  • Components are the unit of reuse - you compose components into more powerful components, into forms, or even into views. The component may be as small as a Button, or as big as a VerticalLayout with a complete form in it.
  • You use Java code to create and nest components; even better you can use Kotlin code to build your UIs in a DSL fashion. You structure the code as you see fit; no need to have 123213 layout XMLs for 45 screen sizes in one large folder.
  • You use a single file CSS to style your app - you don't have to analyze Android Theme shattered into 3214132 XML files.
  • You don't need Emulator nor a device - the browser can render the page as if in a mobile phone; you can switch between various resolutions in an instant.
  • No DEX compilation. The compilation is fast.
  • No fragmentation: you develop for Chrome and Firefox. You can ignore the Internet Explorer crap.
  • No need to fetch stuff in background thread. You actually have two UI threads: the Vaadin UI thread lives server-side, while the browser has its own UI thread; those two threads are completely separated. It is quite typical for a Vaadin app to block in the UI code until the data is fetched, since blocking server-side does not block the UI in the browser.
  • PWAs do not have to be installed. Your users can simply browse your site; if they like the app, they can use their browser to easily create a shortcut to your app on their home screen.
  • You will support both Android and iPhone with one code base.
  • Avoid the trouble of publishing your app on the app stores: the app does not need to be reviewed by anybody; you will not need to pay $100 yearly to Apple.
  • No data syncing necessary since the data will reside on the server.
  • Android's way of requesting for runtime permissions is so horrible: it will call back not your Runnable, but your Activity/Fragment. That disallows you from spliting your code as you see fit; instead you have to keep all of your logic in an Activity/Fragment, shattered because of Fragment-based method callbacks. And since runtime permissions will be required for all Android apps at Google Play at the end of 2018, instead of having to port your app to runtime permissions you can move to another platform.
  • You can find more components at https://vaadin.com/directory. Since Vaadin 10 uses Web Components, you can find even more components at https://www.webcomponents.org/ which can then be turned into Vaadin 10 components simply by implementing the server-side code.

Of course there is no silver bullet. This approach has the following disadvantages:

  • No access to native APIs - only the browser-provided APIs are available. While you can access GPS, you can't access for example the SD card (with local user photos), can't make calls etc.
  • No offline mode unless you develop your app fully or in partial in JavaScript, as a part of the service worker.
  • Vaadin app lives server-side and uses a browser. Because of that, you won't be able to achieve the same performance as with the native app. It's probably not a good fit for, say, an immersive 3D game since the browser is obviously slower than the native code. However, this approach is perfectly fine for a chat client (say, Slack), or a banking app or the like.
  • You will need a SSL certificate since PWAs only work over https
  • You will need to pay for cloud hosting, SSL certificate and DNS domain yourself. The most easy way to set up all that is with Heroku; you can also pay for a virtualized Linux server, get the SSL certificate for free from Let's Encrypt and setup Tomcat to host your app.
  • You are responsible for your app's security; when your app is hacked, all of your user data may be compromised.
  • All of your users will use the same VM; 10000+ users can kill your server unless you load-balance. Luckily it's quite easy with most good cloud providers.

Switching to literally any other platform puts you in charge of code structure. It's as if Google was poised to create a horrible developer hell - eventually you'll develop a Stockholm Syndrome with Android SDK. Just leave that horrible crap behind.

 
7 months ago

In the previous blog post we've used self-signed certificate with Docker Tomcat. However, in order to have a properly protected web site, we need to use a proper set of certificates. We'll use the Let's Encrypt authority to obtain the keys at no cost.

Preparing the server and DNS

First, create a virtual server running Ubuntu 17.10, then make sure you can SSH into that box or you can at least launch a console via the cloud vendor web page. Write down the server IP.

Second, register a DNS domain, for example at GoDaddy. I'll register your-page.eu but you're free to register whatever domain you see fit. You will then need to edit the DNS records for that domain, and make sure that A record points to your server's IP.

Obtaining SSL certificates from Let's Encrypt

Now let's obtain the certificates from Let's Encrypt. The easiest way is to use Certbot which comes pre-packaged in Ubuntu 17.10. Run the following in your box terminal:

sudo apt install certbot
certbot certonly

Certbot is going to ask you a couple of questions. I tend to prefer to use the standalone mode (the certbot takes over port 80 and does everything on its own; this collides with Tomcat listening on 80 but since we need to stop Tomcat to renew the certificates anyway this is perfectly fine). Then just make sure to pass in both your-page.eu and www.your-page.eu so that both domains are SSL-protected.

When Certbot finishes, you'll be able to find the certificate files at /etc/letsencrypt/live/your-page.eu. We will now register the certificates into Tomcat.

Running dockerized Tomcat with the certificates

Let's deploy your app in a way that everything can be started using the docker-compose up -d command. The easiest way is to create a directory named /root/app/ and place the following docker-compose.yml file there:

version: '2'
services:
  web:
    image: tomcat:9.0.5-jre8
    ports:
      - "80:8080"
      - "443:8443"
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - ./server.xml:/usr/local/tomcat/conf/server.xml

Note: we need to map the entire /etc/letsencrypt into the Docker container; just mapping the /etc/letsencrypt/live/your-page.eu folder alone is not enough since those pem files are symlinks which would stop working in Docker and Tomcat would fail with FileNotFoundException.

Feel free to use the server.xml from the self-signed openssl article, but change the appropriate connector part to:

   <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
              maxThreads="150" SSLEnabled="true" >
       <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
         <SSLHostConfig>
       <Certificate certificateKeyFile="/etc/letsencrypt/live/your-page.eu/privkey.pem"
                    certificateFile="/etc/letsencrypt/live/aedict-online.eu/cert.pem"
                    certificateChainFile="/etc/letsencrypt/live/aedict-online.eu/chain.pem"
                    type="RSA" />
            </SSLHostConfig>
    </Connector>

Store the server.xml into /root/app/.

Now go into /root/app and try to run the Tomcat server:

cd /root/app
docker-compose up

The pems are not password-protected so it should work.

Note: if Tomcat stops on Deploying web application directory and seems to do nothing, it may have ran out of entropy. You can verify this by running cat /proc/sys/kernel/random/entropy_avail - if this prints anything less than 100-200 then just run apt install haveged to help building up the entropy pool. Tomcat will take 2 minutes to start at first, but when the entropy is high, further starts will be much quicker. You can read more at Tomcat Hangs.

Now you can visit https://your-page.eu and you should see the Tomcat welcome page protected by https. Good job!

You can also verify at https://www.digicert.com/help/ that your SSL certificate is working properly.

Auto-renewing certificates

Let's Encrypt certificates only last for 90 days. Luckily the certificates can be renewed when they are older than 60 days, which gives us some time to do the renewal process. The certificates can be updated automatically using the Certbot. The Certbot can run hook scripts before and after the renewal happens, so that:

  • we can stop Tomcat in order for the Certbot to take over the port 80,
  • we can start Tomcat to reload certificates when the new certificates are in place

Let's create two scripts: /root/start_server:

#!/bin/bash

set -e -o pipefail

date
cd /root/app
docker-compose up -d

/root/stop_server:

#!/bin/bash

set -e -o pipefail

date
cd /root/app
docker-compose down

Luckily docker-compose down will block until all containers are stopped, and it will unbind from port 80 prior terminating, which gives room for Certbot's own server.

To run the renewal process automatically, run

crontab -e

and add the following line:

30 2 * * 1 certbot renew --pre-hook "/root/stop_server" --post-hook "/root/start_server" >> /var/log/le-renew.log

That will make Certbot to check for new certificates every week at night. Certbot will do nothing if the certificates do not need to be renewed (since they're younger than 60 days). If the certificates need to be renewed (they are older than 60 days) it will stop the Tomcat server, update the certificates and then start the Tomcat server.

That's it.

 
7 months ago

It is possible to use pem-style certificates with Tomcat Docker image, without any need to store them first into the Java keystore. This is excellent since not only it is easier to generate self-signed certificate with the openssl command, this can also be used with certificates produced by Let's Encrypt.

Let's first see how to use the self-signed keys with the Tomcat Docker 9 image. First we generate the self-signed certificate:

$ openssl req -x509 -newkey rsa:4096 -keyout localhost-rsa-key.pem -out localhost-rsa-cert.pem -days 36500

Specify "changeit" as a password (or any other password of your chosing); the Common Name/FQDN is your domain, say, testing.com.

Place the generated localhost-rsa-*.pem files into a ssl/ folder. Now create the following docker-compose.yml file:

version: '2'
services:
  web:
    image: tomcat:9.0.5-jre8
    ports:
      - "8080:8080"
      - "8443:8443"
    volumes:
      - ./ssl:/usr/local/tomcat/ssl
      - ./server.xml:/usr/local/tomcat/conf/server.xml

Place the following server.xml file next to the docker-compose.yml file:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <!-- Security listener. Documentation at /docs/config/listeners.html
  <Listener className="org.apache.catalina.security.SecurityListener" />
  -->
  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container",
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
    -->


    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    -->
    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
         This connector uses the NIO implementation. The default
         SSLImplementation will depend on the presence of the APR/native
         library and the useOpenSSL attribute of the
         AprLifecycleListener.
         Either JSSE or OpenSSL style configuration may be used regardless of
         the SSLImplementation selected. JSSE style configuration is used below.
    -->
    <!--
    <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true">
        <SSLHostConfig>
            <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>
    -->
    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2
         This connector uses the APR/native implementation which always uses
         OpenSSL for TLS.
         Either JSSE or OpenSSL style configuration may be used. OpenSSL style
         configuration is used below.
    -->
    <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="150" SSLEnabled="true" >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
        <SSLHostConfig>
            <Certificate certificateKeyFile="ssl/localhost-rsa-key.pem"
                         certificateFile="ssl/localhost-rsa-cert.pem"
certificateKeyPassword="changeit"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />


    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

      <!-- Use the LockOutRealm to prevent attempts to guess user passwords
           via a brute-force attack -->
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <!-- This Realm uses the UserDatabase configured in the global JNDI
             resources under the key "UserDatabase".  Any edits
             that are performed against this UserDatabase are immediately
             available for use by the Realm.  -->
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

      </Host>
    </Engine>
  </Service>
</Server>

The directory structure should look like this:

.
├── docker-compose.yml
├── server.xml
└── ssl
    ├── localhost-rsa-cert.pem
    └── localhost-rsa-key.pem

That's it - just run docker-compose up in the . folder and Tomcat should boot up. Now you can visit https://localhost:8443 and the browser should warn you about the self-signed certificate - just store an exception for the site and you should see the Tomcat welcome page served over https.

 
8 months ago

I have to admit upfront that my brain is somehow incompatible with JPA. Every time I try to use JPA, I will somehow wind up fighting weird issues, be it LazyInitializationException with Hibernate, or weird sequence generator preallocation issues with EclipseLink. I don't know what to use merge() for; I lack the mean to reattach a bean and overwrite its values from the database. I don't want to bore you to death with all of the nitty-gritty details; all the reasons are summed in Issue #3 Remove JPA. I think that the number 1 reason of why JPA is incompatible with me is that

JPA is built around the idea that the Java classes, not the database, is the source of truth.

My way of thinking is the other way around: I think that the database (and not the JPA) is the source of truth. This way of thinking is however incompatible with the very core of the JPA technology and makes it impossible to use JPA for me.

The Source Of Truth

To allow Java beans to be the source of truth, JPA defines certain techniques and requires Hibernate to sync Java beans with the database behind the scenes. That is unfortunately a non-trivial issue which cannot be done automatically in certain cases, and that's why JPA abstraction suffers from the following issues:

  • Hibernate must track changes done to the bean, in order to sync them to the underlying database. That is done by intercepting all calls to setters, which can only be done by runtime class enhancement (for every entity Hibernate creates a class which extends that entity, overrides all setters and injects some Hibernate core class which is notified of changes). Such class can hardly be called a POJO since passing that class around will start throwing LazyInitializationException like hell. That is the black magic, pure and evil.
  • It intentionally lacks the method to reattach a bean and overwrite its contents with the database contents - this goes against the idea of having Java class the source of truth.
  • By abstracting joins into Collections, it causes the programmers to unintentionally write code which suffers from the 1+N loading issues.

Since I'm very bullheaded, instead of trying to attune my mind towards JPA I explored the different approach: let's step back and make the database the source of truth again.

Database As The Source Of Truth

Having Database as the source of truth means that the data in your bean may be obsolete the moment it leaves the database (since there might be somebody else already modifying that thing). The optimistic/pessimistic locking is the way around that, and to have the most recent data you will need to requery from the database.

The bean can now ditch all of its sync capabilities and become just a snapshot: a sample of the state of the database row at some particular time. Changes to the bean properties will no longer be propagated automatically to the database, but need to be done explicitly by the developer. The developer is once again in charge.

There - I have solved the whole detached/managed/lazy initialization exception nonsense for you :-)

The beans are once again POJOs with getters and setters, simply populated from the database by reading the data from the JDBC ResultSet and calling bean's setters.

That can be achieved by a simple reflection. Moreover, no runtime enhancement is needed since the bean will never track changes; to save changes we will need to issue the INSERT statement. Very important thought: since there's no runtime enhancement, the bean is really a POJO and can be actually made serializable and passed around in the app freely; you don't need DTOs anymore.

The Vaadin-on-Kotlin Persistency Layer: vok-db

There is a very handy library called Sql2o which maps JDBC ResultSet to Java POJOs. This idea forms the very core of vok-db. To explain the data retrieving capabilities of the vok-db library, let's begin by implementing a simple function which finds a bean by its ID.

vok-db: Data Retrieval From Scratch

The findById() function will simply issue a select SQL statement which selects a single row, then passes it to Sql2o to populate the bean with the contents of that row:

fun <T : Any> Connection.findById(clazz: Class<T>, id: Any): T? =
        createQuery("select * from ${clazz.databaseTableName} where id = :id")
                .addParameter("id", id)
                .executeAndFetchFirst(clazz)

The function is rather simple: it is actually nothing more but a simple Kotlin reimplementation of the very first "Execute, fetch and map" example in the Sql2o home page. The Sql2o API will do nothing more but create a JDBC PreparedStatement, run it, create instances of clazz using the zero-arg constructor, populate the bean with data and return the first one. Using this approach it is easy to implement more of the finders, such as findAll(clazz: Class<T>), findBySQL(clazz: Class<T>, where: String). However, let's focus on the findById() for now.

To obtain the Sql2o's Connection (which is a wrapper of JDBC's connection), we simply poll the HikariCP connection pool and use the db{} method to start+commit the transaction for us. This will allow us to retrieve beans from the database easily, simply by calling the following statement from anywhere of your code, be it a background thread, or a Vaadin button click listener:

val person: Person = db { connection.findById(Person::class.java, 25L) }

This is nice, but we can do better. In the dependency injection world, for every bean there is a Repository or DAO class which defines such finder methods and fetches instances of that particular bean. However, we have no DI and therefore we do not have to have a separate class injected somewhere, just for the purpose of accessing a database. We are free to attach those finders right where they belong: as factory methods of the Person class:

class Person {
  companion object {
    fun findById(id: Long): Person? = db { connection.findById(Person::class.java, id) }
  }
}

This will allow us to write the finder code even more concise:

val person = Person.findById(25L)

Yet, adding those pesky findById() (and all other finder methods such as findAll()) manually to every bean is quite a tedious process. Is there perhaps some easier, more automatized way to add those methods to every bean? Yes there is, by the means of mixins and companion objects.

As a first attempt, we will define a Dao interface as follows:

interface Dao<T> {
  fun findById(clazz: Class<T>, id: Any): T? = db { connection.findById(clazz, id) }
}

Remember that the companion object is just a regular object, and as such it can implement interfaces?

class Person {
  companion object : Dao<Person>
}

This allows us to call the finder method as follows:

val person = Person.findById(Person::class.java, 25L)

Not bad, but the Person::class.java is somewhat redundant. Is there a way to force Kotlin to fill it for us? Yes, by the means of extension methods and reified type parameters:

interface Dao<T> {}
inline fun <ID: Any, reified T : Entity<ID>> Dao<T>.findById(id: ID): T? = db { con.findById(T::class.java, id) }

Now the finder code is perfect:

val person = Person.findById(25L)

vok-db follows this idea and provides the Dao interface for you, along with several of utility methods for your convenience. By the means of extension methods, you can define your own global finder methods, or you can simply attach finder methods to your beans as you see fit.

You can even define an extension property on Dao which produces Vaadin's DataProvider, fetching all instances of that particular bean. Luckily, you don't have to do that since vok-db already provides one for you.

Next up, data modification.

vok-db: Data Modification

Unfortunately Sql2o doesn't support persisting of the data, so we will have to do that ourselves. Luckily, it's very simple: all we need to do is to generate a proper insert or update SQL statement:

interface Entity<ID: Any> : Serializable {
    ...
    fun save() {
        db {
            if (id == null) {
                // not yet in the database, insert
                val fields = meta.persistedFieldDbNames - meta.idDbname
                con.createQuery("insert into ${meta.databaseTableName} (${fields.joinToString()}) values (${fields.map { ":$it" }.joinToString()})")
                        .bind(this@Entity)
                        .executeUpdate()
                id = con.key as ID
            } else {
                val fields = meta.persistedFieldDbNames - meta.idDbname
                con.createQuery("update ${meta.databaseTableName} set ${fields.map { "$it = :$it" }.joinToString()} where ${meta.idDbname} = :${meta.idDbname}")
                        .bind(this@Entity)
                        .executeUpdate()
            }
        }
    }
    ...
}

Simply by having your beans implement the Entity interface, you will gain the ability to save and/or create database rows:

data class Person(override var id: Long, var name: String): Entity<Long> {
  companion object : Dao<Person>
}

val person = Person.findById(25L)
person.name = "Duke Leto Atreides"
person.save()

Pure simplicity: