Wrong!Condition failed with Exception:transformer.
toEmployeeProto(expected) == input| | || | Employee(name=bob, salary=45, bonus=null)| java.
lang.
NullPointerExceptioncom.
efenglu.
jprotoc.
example.
EmployeeTransformer@7c83dc97Hmm, that’s weird.
Let’s see, when the bonus was null we got a NPE because we called set with a null.
This is because:Protobuf does not allow nullsWell it does and it doesn’t.
Protobuf does not allow you to set a field to null.
It will throw an NPE on calling set with a null.
Well, what about our other spec:def "test toEmployee"(EmployeeProto input, Employee expected) { given: EmployeeTransformer transformer = new EmployeeTransformer() expect: transformer.
toEmployee(input) == expected where: input | expected createEmployeeProto("bob", 45, 100) | createEmployee("bob", 45, 100) createEmployeeProto("bob", 45, 0) | createEmployee("bob", 45, 0) createEmployeeProto("bob", 45, null) | createEmployee("bob", 45, null)}How did this one fair:Condition not satisfied:transformer.
toEmployee(input) == expected| | | | || | | | Employee(name=bob, salary=45, bonus=null)| | | false| | name: "bob"| | salary {| | currency_code: "USD"| | units: 45| | }| Employee(name=bob, salary=45, bonus=0)com.
efenglu.
jprotoc.
example.
EmployeeTransformer@198b6731We were expecting a null value but got back 0.
Why?Protobuf does not allow nullsProtobuf will return a default object when a field has NOT been set.
Ergo, the value is 0 instead of null.
https://xkcd.
com/386/Again this article is not meant to discuss the merits of the use of nulls/non nulls etc.
Lets move beyond and figure out how to deal with the cards we’ve been delt.
We have a domain model that is expecting nullsWe have a transport framework that does not allow nullsOn the surface this seems to be an intractable problem.
But not entirely.
While protobuf doesn’t directly support the concept on null it does provide a ‘has’ concept.
The has boolean is meant to tell you if a value has been set.
We can use this to transport our concept of null.
Let’s modify our code to use has and avoid the NPE’s.
public class EmployeeTransformerHas { public Employee toEmployee(EmployeeProto employeeProto) { final Employee.
EmployeeBuilder builder = Employee.
newBuilder() .
name(employeeProto.
getName()); if (employeeProto.
hasSalary()) { builder.
salary(toBigDecimal(employeeProto.
getSalary())); } if (employeeProto.
hasBonus()) { builder.
bonus(toBigDecimal(employeeProto.
getBonus())); } return builder.
build(); } public EmployeeProto toEmployeeProto(Employee employee) { final EmployeeProto.
Builder builder = EmployeeProto.
newBuilder() .
setName(employee.
getName()); if (employee.
getSalary() != null) { builder.
setSalary(toMoney(employee.
getSalary())); } if (employee.
getBonus() != null) { builder.
setBonus(toMoney(employee.
getBonus())); } return builder.
build(); } .
}While our tests now pass, the transformer code is quite ugly.
Also, while the transformer provides help for existing domain objects we would really prefer if our clients spoke protobuf natively.
Once our programmers start utilizing the code they will have to write all of these checks by hand.
It would be nice if we could somehow support null on set and null on get.
Enter the protoc plugin.
Protoc Plugin Extension PointProtoc has an extension point mechanism to allow you to insert code along side generated code.
This will allow us to add helper/utility methods that have the behavior we want.
First some bad news.
Without directly editing the original C code for the protoc compiler you can’t change any existing generated code.
So you can’t directly change the set method to accept null, or the get method to return null.
What you can do is add new methods to the generated classes.
Like adding a setOrClear method and a nullGet method.
For our project we decided to add a setOrClear and an optionalGet method.
setOrClearProvides a setOrClear method to the generated builder which accepts a null value.
If the value provided is null the builder will call clear on the associated field.
If the value provided is non-null the builder will call set on the associated field with the provided value.
This avoids the NPE that is normally thrown when set is called with a null value.
Example:public Builder setOrClearBonus(com.
google.
type.
Money value) { if (value != null) { return setBonus(value); } else { clearBonus(); } return this;}optionalGetAdds a method optional{Field} that will wrap all non primitivefields of a message type in a java.
util.
Optional object.
When has{Field} returns false the optional is emptyWhen has{Field} returns true the optional contains the value of calling get{Field}.
Example:public java.
util.
Optional<com.
google.
type.
Money> optionalBonus() { if (hasBonus()) { return java.
util.
Optional.
of(getBonus()); } else { return java.
util.
Optional.
empty(); }}With the combination of setOrClear and optional we felt this was the best of both worlds.
We can set nulls and still provide non null return types.
But…Some other bad news.
This approach requires developers to call the correct methods.
set still throws NPE, and get still returns a default value.
This isn’t ideal but could also be useful for those used to the expected Protobuf behavior.
Also these new methods ONLY apply to proto’s that you have compiled.
So if you’re reusing someone elses prepackaged proto definitions you won’t get these new methods.
Creating the PluginHow do we write this magical plugin.
You could write it in C and use some of the existing protoc library code.
However, we are a Java shop and maintaining C code wasn’t really an option.
Thankfully, protoc doesn’t actually care what your plugin is written in.
All protoc does is invoke the plugin executable and pass in a protobuf string describing the protobuf we want to generate.
Very meta, also very cool.
Salesforce helpfully provides a maven plugin to assist our effort.
This allows us to write our plugin in Java as a regular maven artifact.
A link to the full codebase is found at the end of the article but lets focus on a couple pieces first.
Step 1: Setup maven moduleNothing really special here.
A regular maven jar module is all we need.
Just make sure we depend on Salesforce jprotoc<dependency> <groupId>com.
salesforce.
servicelibs</groupId> <artifactId>jprotoc</artifactId> <version>0.
9.
0</version></dependency>Step 2: Create your extension point main classAs we mentioned earlier protoc treats all extensions as executable programs we therefore need to create a Java class with a standard main entry point:public class JProtoc extends com.
salesforce.
jprotoc.
Generator { public static void main(String[] args) { com.
salesforce.
jprotoc.
ProtocPlugin.
generate(new JProtoc()); } .
}Simple enough so far.
Next override the super generate method:generateWe are returning a stream of File objects.
These represent files that are to be generated by protocWe are filtering it by checking if the proto to generate is in the list of files to generate.
You will be passed all the proto files, even imported ones, so be sure to filter these out.
(Unless you want to generate code for them)We then call our other method to generate our code in handleProtoFilehandleProtoFileNothing too fancy here.
Just be aware that the java package could come from two different locations.
Either implicitly equal to the proto package or explicitly with the proto option.
handleMessageTypeNow we’re getting into the real meat.
First build the filename of the class we are going to be adding code to.
This isn’t really correct here since I’m assuming you are using the multiple file option.
Without this all proto classes and inner classes and should be delt with differently.
We then call other sub-methods to build the strings for the code we are inserting.
It’s nothing really special, just inserting values into a mustache template.
Checkout the full code for details.
The real kicker here is how we generate the file object.
Notice how we set the fileName, the content we want to insert and the insertion point.
The insertion point is a special string in a comment that shows up in the protoc generated code.
There are several different insertion points.
The ones we are using are the builder_scope and the class_scope insertion points.
Example:// @@protoc_insertion_point(class_scope:com.
efenglu.
protobuf.
EmployeeProto)builder_scopeInserts code at the end of the inner builder class.
This is where we want to add our setOrClear methods.
class_scopeInserts code at the end of the java class for the message type.
This is where we want to add our optionalGet methods.
Other insertion points exist.
Open the generated proto files and search for protoc_insertion_point to see what’s available.
Step 3: Use the PluginUsing the new plugin is quite easy.
Configure the maven project to build the proto files using the Maven Proto Buffers Plugin:Maven Protocol Buffers Plugin – IntroductionMaven Protocol Buffers Plugin uses Protocol Buffer Compiler ( protoc) tool to generate Java source files from .
proto…www.
xolstice.
orHere is how we hook in our new plugin:os-maven-plugin: Helps us in downloading the correct protoc binary for our build systemprotobuf-maven-plugin: Hooks in the protoc binary and our custom protoc pluginNotice the protocPlugin element.
Here we’ve hooked in our protoc plugin.
We specify the maven artifact for our plugin and the executable class within that artifact.
The protobuf-maven-plugin has excellent docs exampling the usage in greater detail.
With this we have our new custom methods.
With great power…You can basically add whatever you want into your generated proto code using this approach.
But there are severals things to consider before doing so.
This is only added to your codeThis is not what someone would expectCould you do it some other way?I hope this has been helpful.
There are several other challanges to using the protoc plugin framework, such as adding comments from the proto files, dealing with oneof, maps and repeated types, fields name collisions.
But this should get you started.
For the complete source code referenced in this article check the Github repo:efenglu/jprotocSample use case of Protoc Plugin written in Java.
Contribute to efenglu/jprotoc development by creating an account on…github.
com.. More details