Looks good for a beginner.
Couple tweaks if you want to make it look nicer. The name of the class should be somewhat describing the purpose so in your case
Calculator would fit better. If you are familiar with
enum you could parse your
/ into an enum and do
switch on that. Then next tweak could be to do the
System.printout at the end and in the switch case to do just the operation.
Another suggestion would be to make 4 classes implementing binary operations with just one method taking two arguments and returning the result. Then you can have a map with a key being the operation (in enum or string) and the value would be the appropriate class. Then you end up with more classes but this
main method gets shorter.
Generally it is better to split the code in more methods and classes and keep the motto "one class one responsibility", or Single Responsibility Principle. So as an example each of your arithmetic classes would be responsible for a single arithmetic operation. If you want to follow that completely, then you would have to create a class that gets the input from the user as well. The Calculator class then would be just like a coordinator saying: give me numbers, give me an operator, perform the operation and print out the result at the end.
Note: If you want some sample code leave me a comment here and I can add some.